[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496810749 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003307] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008834] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011144] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013043] SELinux: Initializing. [ 0.014069] *** VALIDATE selinux *** [ 0.022614] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027160] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030121] *** VALIDATE tmpfs *** [ 0.032342] *** VALIDATE proc *** [ 0.033235] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035270] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037103] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039033] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043028] debug: unmapping init [mem 0xffffffffa9459000-0xffffffffa9460fff] [ 0.045960] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046671] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047024] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051012] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053011] ... event mask: 000000070000000f [ 0.054291] rcu: Hierarchical SRCU implementation. [ 0.056350] smp: Bringing up secondary CPUs ... [ 0.057634] x86: Booting SMP configuration: [ 0.058026] .... node #0, CPUs: #1 #2 #3 [ 0.062013] smp: Brought up 1 node, 4 CPUs [ 0.064020] smpboot: Max logical packages: 1 [ 0.065013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.225314] node 0 deferred pages initialised in 157ms [ 0.229154] devtmpfs: initialized [ 0.230298] x86/mm: Memory block size: 128MB [ 0.235888] gcov: version magic: 0x41383552 [ 0.239480] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.244128] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.247780] pinctrl core: initialized pinctrl subsystem [ 0.250458] [ 0.251010] ************************************************************* [ 0.254019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.257022] ** ** [ 0.259017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.262020] ** ** [ 0.264017] ** This means that this kernel is built to expose internal ** [ 0.266019] ** IOMMU data structures, which may compromise security on ** [ 0.269020] ** your system. ** [ 0.271016] ** ** [ 0.273018] ** If you see this message and you are not debugging the ** [ 0.276019] ** kernel, report this immediately to your vendor! ** [ 0.278016] ** ** [ 0.280018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.283019] ************************************************************* [ 0.285858] NET: Registered protocol family 16 [ 0.288775] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.291092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.294122] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.298135] cpuidle: using governor menu [ 0.299852] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.302539] PCI: Using configuration type 1 for base access [ 0.304120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.313103] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.316033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.319034] cryptd: max_cpu_qlen set to 1000 [ 0.321354] ACPI: Added _OSI(Module Device) [ 0.323016] ACPI: Added _OSI(Processor Device) [ 0.324011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.325017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.331668] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.338363] ACPI: Interpreter enabled [ 0.340067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.341015] ACPI: Using IOAPIC for interrupt routing [ 0.343096] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.347435] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.359025] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.361043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.364018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.367082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.372487] acpiphp: Slot [2] registered [ 0.374167] acpiphp: Slot [5] registered [ 0.376162] acpiphp: Slot [6] registered [ 0.377191] acpiphp: Slot [7] registered [ 0.379142] acpiphp: Slot [8] registered [ 0.380158] acpiphp: Slot [9] registered [ 0.382123] acpiphp: Slot [10] registered [ 0.383122] acpiphp: Slot [3] registered [ 0.384092] acpiphp: Slot [4] registered [ 0.385100] acpiphp: Slot [11] registered [ 0.386082] acpiphp: Slot [12] registered [ 0.388135] acpiphp: Slot [13] registered [ 0.390257] acpiphp: Slot [14] registered [ 0.392177] acpiphp: Slot [15] registered [ 0.394140] acpiphp: Slot [16] registered [ 0.396149] acpiphp: Slot [17] registered [ 0.397167] acpiphp: Slot [18] registered [ 0.399164] acpiphp: Slot [19] registered [ 0.401266] acpiphp: Slot [20] registered [ 0.403229] acpiphp: Slot [21] registered [ 0.404135] acpiphp: Slot [22] registered [ 0.406140] acpiphp: Slot [23] registered [ 0.408164] acpiphp: Slot [24] registered [ 0.409122] acpiphp: Slot [25] registered [ 0.411113] acpiphp: Slot [26] registered [ 0.412129] acpiphp: Slot [27] registered [ 0.414134] acpiphp: Slot [28] registered [ 0.415103] acpiphp: Slot [29] registered [ 0.416176] acpiphp: Slot [30] registered [ 0.418122] acpiphp: Slot [31] registered [ 0.419103] PCI host bridge to bus 0000:00 [ 0.420038] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.423039] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.425035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.428037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.431037] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.434039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.436296] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.440217] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.443393] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.454801] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.460687] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.463033] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.466028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.468035] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.471907] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.474854] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.477056] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.479881] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.485020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.497023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.502023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.507000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.514019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.519015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.535024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.544431] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.551019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.556013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.574025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.587128] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.595025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.602021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.625018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.634668] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.642020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.647018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.662023] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.672504] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.679016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.687017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.711017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.720813] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.728018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.736021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.753021] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.760858] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.764421] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.767350] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.769395] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.771242] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.776038] iommu: Default domain type: Passthrough [ 0.777468] SCSI subsystem initialized [ 0.779162] ACPI: bus type USB registered [ 0.780083] usbcore: registered new interface driver usbfs [ 0.781128] usbcore: registered new interface driver hub [ 0.783089] usbcore: registered new device driver usb [ 0.785187] pps_core: LinuxPPS API ver. 1 registered [ 0.787013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.790069] PTP clock support registered [ 0.792127] EDAC MC: Ver: 3.0.0 [ 0.794132] PCI: Using ACPI for IRQ routing [ 0.796870] NetLabel: Initializing [ 0.798018] NetLabel: domain hash size = 128 [ 0.799011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.801116] NetLabel: unlabeled traffic allowed by default [ 0.804091] vgaarb: loaded [ 0.805273] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.807015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.811342] clocksource: Switched to clocksource kvm-clock [ 0.920714] VFS: Disk quotas dquot_6.6.0 [ 0.922240] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.925029] *** VALIDATE ramfs *** [ 0.926335] *** VALIDATE hugetlbfs *** [ 0.927947] pnp: PnP ACPI init [ 0.930309] pnp: PnP ACPI: found 6 devices [ 0.958877] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.962243] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.964475] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.966726] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.969179] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.971753] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.974778] NET: Registered protocol family 2 [ 0.977391] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.982480] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.986304] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.992866] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.996852] TCP: Hash tables configured (established 65536 bind 65536) [ 1.000284] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.004655] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.007738] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.010442] NET: Registered protocol family 1 [ 1.012886] RPC: Registered named UNIX socket transport module. [ 1.015015] RPC: Registered udp transport module. [ 1.017688] RPC: Registered tcp transport module. [ 1.019305] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.021590] NET: Registered protocol family 44 [ 1.023127] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.025147] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.027118] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.029325] PCI: CLS 0 bytes, default 64 [ 1.030907] Unpacking initramfs... [ 2.466539] debug: unmapping init [mem 0xffff8ca57cc54000-0xffff8ca57ffbffff] [ 2.473306] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.475805] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.480360] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.988301] Initialise system trusted keyrings [ 2.990138] Key type blacklist registered [ 2.992478] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.002628] zbud: loaded [ 3.006299] *** VALIDATE nfs *** [ 3.007603] *** VALIDATE nfs4 *** [ 3.009677] pstore: using deflate compression [ 3.013186] Platform Keyring initialized [ 3.119359] NET: Registered protocol family 38 [ 3.120879] Key type asymmetric registered [ 3.122291] Asymmetric key parser 'x509' registered [ 3.124207] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.127252] io scheduler mq-deadline registered [ 3.128873] io scheduler kyber registered [ 3.130473] io scheduler bfq registered [ 3.133052] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.135787] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.138392] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.140866] ACPI: Power Button [PWRF] [ 3.146423] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.154535] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.171301] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.179355] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.197042] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.227295] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.261507] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.268990] Non-volatile memory driver v1.3 [ 3.270654] Linux agpgart interface v0.103 [ 3.303238] virtio_blk virtio1: [vda] 146960 512-byte logical blocks (75.2 MB/71.8 MiB) [ 3.305931] vda: detected capacity change from 0 to 75243520 [ 3.323822] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.326502] vdb: detected capacity change from 0 to 1073741824 [ 3.342769] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.345655] vdc: detected capacity change from 0 to 2621440000 [ 3.360522] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.363863] vdd: detected capacity change from 0 to 2621440000 [ 3.381994] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.385032] vde: detected capacity change from 0 to 4294967296 [ 3.403201] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.405964] vdf: detected capacity change from 0 to 4294967296 [ 3.415729] libphy: Fixed MDIO Bus: probed [ 3.422517] usbcore: registered new interface driver usbserial_generic [ 3.424203] usbserial: USB Serial support registered for generic [ 3.426921] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.431556] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.433551] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.438408] mousedev: PS/2 mouse device common for all mice [ 3.441504] rtc_cmos 00:05: RTC can wake from S4 [ 3.444659] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.444797] rtc_cmos 00:05: registered as rtc0 [ 3.449671] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.452731] intel_pstate: CPU model not supported [ 3.456113] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.458207] hid: raw HID events driver (C) Jiri Kosina [ 3.462119] usbcore: registered new interface driver usbhid [ 3.464357] usbhid: USB HID core driver [ 3.464516] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.465936] drop_monitor: Initializing network drop monitor service [ 3.471560] Initializing XFRM netlink socket [ 3.473484] NET: Registered protocol family 10 [ 3.475947] Segment Routing with IPv6 [ 3.477071] NET: Registered protocol family 17 [ 3.478937] mpls_gso: MPLS GSO support [ 3.483633] RAS: Correctable Errors collector initialized. [ 3.485351] AVX version of gcm_enc/dec engaged. [ 3.487123] AES CTR mode by8 optimization enabled [ 3.569857] sched_clock: Marking stable (3569834421, 0)->(4499309604, -929475183) [ 3.575104] registered taskstats version 1 [ 3.577937] Loading compiled-in X.509 certificates [ 3.580252] zswap: loaded using pool lzo/zbud [ 3.603509] Key type big_key registered [ 3.618682] Key type encrypted registered [ 3.620151] ima: No TPM chip found, activating TPM-bypass! [ 3.622261] ima: Allocated hash algorithm: sha1 [ 3.623915] ima: No architecture policies found [ 3.625598] evm: Initialising EVM extended attributes: [ 3.627582] evm: security.selinux [ 3.628643] evm: security.ima [ 3.629438] evm: security.capability [ 3.630604] evm: HMAC attrs: 0x1 [ 3.632570] rtc_cmos 00:05: setting system clock to 2026-08-27 22:05:46 UTC (1787868346) [ 3.639071] debug: unmapping init [mem 0xffffffffaa403000-0xffffffffaa5fffff] [ 3.642023] debug: unmapping init [mem 0xffffffffa9182000-0xffffffffa9458fff] [ 3.651171] Write protecting the kernel read-only data: 28672k [ 3.654707] debug: unmapping init [mem 0xffffffffa7803000-0xffffffffa79fffff] [ 3.657542] debug: unmapping init [mem 0xffffffffa8114000-0xffffffffa81fffff] [ 3.691180] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.699612] systemd[1]: Detected virtualization kvm. [ 3.701517] systemd[1]: Detected architecture x86-64. [ 3.703396] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.727392] systemd[1]: No hostname configured. [ 3.729130] systemd[1]: Set hostname to . [ 3.731304] random: systemd: uninitialized urandom read (16 bytes read) [ 3.733783] systemd[1]: Initializing machine ID from random generator. [ 3.772175] random: ln: uninitialized urandom read (6 bytes read) [ 3.882267] random: systemd: uninitialized urandom read (16 bytes read) [ 3.886753] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.896656] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.903819] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.589637] device-mapper: uevent: version 1.0.3 [ 4.591731] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.341961] virtio_net virtio0 ens2: renamed from eth0 [ 5.347768] random: fast init done [ 5.359271] scsi host0: ata_piix [ 5.378240] scsi host1: ata_piix [ 5.379973] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.382443] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.692943] dracut-initqueue[584]: RTNETLINK answers: File exists [ 10.138093] random: crng init done [ 10.139960] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.538240] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.672986] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.950388] SELinux: Disabled at runtime. [ 12.014688] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.023927] systemd[1]: Detected virtualization kvm. [ 12.025973] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.512923] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.516536] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.521929] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.526710] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.531521] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.539137] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.544948] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ 12.672544] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.953172] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.263547] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.332968] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.494710] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.514050] EDAC sbridge: Ver: 1.1.2 [ 17.125570] Key type dns_resolver registered [ 17.878848] NFS: Registering the id_resolver key type [ 17.882230] Key type id_resolver registered [ 17.887774] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (5s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Started Login Service. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg417-server login: [ 57.770018] hrtimer: interrupt took 3009533 ns [ 79.407213] libcfs: loading out-of-tree module taints kernel. [ 79.481669] Key type ._llcrypt registered [ 79.496073] Key type .llcrypt registered [ 79.634460] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 98.027437] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 99.915080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 99.929172] alg: No test for adler32 (adler32-zlib) [ 101.668805] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 102.734067] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 104.544451] Key type lgssc registered [ 106.426559] Lustre: Echo OBD driver; http://www.lustre.org/ [ 124.819932] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 172.589467] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 190.480529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 190.531224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 192.043502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 192.105658] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 192.269638] Lustre: lustre-MDT0000: new disk, initializing [ 192.480551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.507824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 198.660280] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 213.323029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 213.478446] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 213.515815] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 213.529978] Lustre: Skipped 1 previous similar message [ 213.709085] Lustre: lustre-MDT0001: new disk, initializing [ 213.818934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 213.858196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 213.875601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 218.878663] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.287072] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 234.954827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 235.262769] Lustre: lustre-OST0000: new disk, initializing [ 235.267573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 235.273400] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 235.345036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 237.139515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 237.146059] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 237.209181] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 241.600396] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 255.663561] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 255.806350] Lustre: lustre-OST0001: new disk, initializing [ 255.811334] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 255.818987] Lustre: 9527:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 255.865599] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 262.224635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 262.239091] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 262.336792] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 264.029519] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.689191] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 284.559374] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 292.228799] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing check_logdir /tmp/testlogs/ [ 298.513725] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing yml_node [ 304.381903] Lustre: DEBUG MARKER: Client: 2.17.57.99 [ 307.531916] Lustre: DEBUG MARKER: MDS: 2.17.57.99 [ 310.782186] Lustre: DEBUG MARKER: OSS: 2.17.57.99 [ 312.700828] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 18:10:53 EDT 2026 [ 332.772250] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 334.576650] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 352.229364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 352.246324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 352.276696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.786358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.788060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 354.801379] Lustre: Skipped 1 previous similar message [ 354.837890] Lustre: Skipped 3 previous similar messages [ 357.788317] Lustre: server umount lustre-MDT0000 complete [ 362.061291] LustreError: 8450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868704 with bad export cookie 3132575018134273989 [ 362.064670] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 362.073760] LustreError: 8450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 365.025382] LustreError: 8568:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.025872] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.063420] LustreError: 8568:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 365.109506] Lustre: Skipped 1 previous similar message [ 365.118856] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 365.141842] Lustre: Skipped 1 previous similar message [ 367.075350] LustreError: 6518:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.105356] LustreError: 6518:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 368.903426] Lustre: server umount lustre-MDT0001 complete [ 373.612367] Lustre: server umount lustre-OST0000 complete [ 378.379766] Lustre: server umount lustre-OST0001 complete [ 386.862885] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 397.819372] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 437.026600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.249288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 437.278091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 437.343441] Lustre: lustre-MDT0000: new disk, initializing [ 437.412127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.435527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 442.382262] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 453.974193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 454.146991] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 454.204460] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 454.208087] Lustre: Skipped 1 previous similar message [ 454.265844] Lustre: lustre-MDT0001: new disk, initializing [ 454.349598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 454.399914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 454.417160] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 459.048074] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 470.656987] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 477.897499] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 486.359527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 486.625104] Lustre: lustre-OST0000: new disk, initializing [ 486.628413] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 486.638675] Lustre: 17052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 486.790610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 488.132455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 488.151900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 488.349133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 493.011362] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 503.118825] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 509.028345] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 509.265756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 514.309103] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 514.632557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 519.144155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 519.150985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 519.164915] LustreError: Skipped 1 previous similar message [ 519.170057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 519.170068] Lustre: Skipped 2 previous similar messages [ 519.208099] Lustre: Skipped 2 previous similar messages [ 523.221312] Lustre: server umount lustre-OST0000 complete [ 529.394030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.418669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 529.427964] Lustre: Skipped 2 previous similar messages [ 533.733374] Lustre: server umount lustre-MDT0000 complete [ 534.498600] LustreError: 17033:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.533850] LustreError: 17033:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 538.123940] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787868880 with bad export cookie 3132575018134275515 [ 538.134180] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.135090] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 538.572493] Lustre: server umount lustre-MDT0001 complete [ 545.239214] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 18:14:46 (1787868886) [ 565.844430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 566.589828] LustreError: 19927:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 566.693340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.875287] LustreError: 19928:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.275416] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 581.890879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 582.117647] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 587.573704] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 596.529982] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 602.788511] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 605.225021] Lustre: Failing over lustre-MDT0000 [ 605.464958] Lustre: server umount lustre-MDT0000 complete [ 607.273253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 607.301360] Lustre: Skipped 1 previous similar message [ 607.319504] LustreError: 19921:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 607.333485] LustreError: 19921:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 623.483048] Lustre: 3641:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787868950/real 1787868950] req@ffff8ca4c3a5ed00 x1874715942750848/t0(0) o400->MGC192.168.204.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787868966 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 623.537306] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 623.553706] LustreError: 19927:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 623.579500] LustreError: 19927:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 626.828084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.828705] LustreError: 3640:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8ca5f49f70c0 x1874715942754048/t0(0) o250->MGC192.168.204.117@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 634.146075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.156664] Lustre: Skipped 1 previous similar message [ 634.226836] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 639.457975] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 639.500571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 639.527415] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 639.867713] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 650.994241] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 657.891575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 657.911071] Lustre: Skipped 1 previous similar message [ 657.915558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 657.919326] Lustre: Skipped 1 previous similar message [ 661.395256] Lustre: server umount lustre-MDT0000 complete [ 663.013405] LustreError: 19923:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 663.034345] LustreError: 19923:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 665.990322] LustreError: 19907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869008 with bad export cookie 3132575018134276936 [ 666.011115] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 666.014471] LustreError: 19907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 666.522789] Lustre: server umount lustre-MDT0001 complete [ 676.055529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 681.953109] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 692.160919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 697.983750] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 707.174779] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 713.323383] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 715.143202] Lustre: Failing over lustre-MDT0000 [ 715.392285] Lustre: server umount lustre-MDT0000 complete [ 717.223574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 717.231772] Lustre: Skipped 2 previous similar messages [ 727.522160] LustreError: 23262:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 727.547717] LustreError: 23262:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 733.664116] Lustre: 3642:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787869060/real 1787869060] req@ffff8ca5d4969a40 x1874715942781440/t0(0) o400->MGC192.168.204.117@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787869076 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 733.700327] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 735.800881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 745.302873] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 750.038451] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 750.562263] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 750.575418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 750.591495] Lustre: Skipped 1 previous similar message [ 750.612087] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 760.307441] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 765.921891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 765.932454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 770.720261] Lustre: server umount lustre-MDT0000 complete [ 775.241588] LustreError: 23248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869118 with bad export cookie 3132575018134277874 [ 775.256546] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 775.703796] Lustre: server umount lustre-MDT0001 complete [ 784.586344] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 18:18:45 (1787869125) [ 804.007728] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 813.736399] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 858.145535] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 870.615841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 870.929482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 870.985191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 871.097286] Lustre: lustre-MDT0000: new disk, initializing [ 871.264888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 871.272327] Lustre: Skipped 3 previous similar messages [ 871.303532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 876.661950] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 888.622178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.742837] Lustre: 29645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 888.970612] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 894.616559] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 906.565742] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 913.591504] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 921.896538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 922.322923] Lustre: 31617:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 924.175233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 924.245806] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 924.337290] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 924.353023] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 928.336983] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 937.986781] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 945.838151] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 970.210054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 970.215115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.237585] Lustre: Skipped 2 previous similar messages [ 970.247765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 970.256801] Lustre: Skipped 2 previous similar messages [ 975.853368] LustreError: 31603:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 975.863044] LustreError: 31603:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 15 previous similar messages [ 975.955245] Lustre: server umount lustre-OST0000 complete [ 980.970166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 980.986577] LustreError: Skipped 1 previous similar message [ 986.543883] Lustre: server umount lustre-MDT0000 complete [ 990.401434] LustreError: 29637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869333 with bad export cookie 3132575018134278280 [ 990.407781] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 990.421088] LustreError: 29637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 990.832754] Lustre: server umount lustre-MDT0001 complete [ 1003.613497] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 1006.683221] Key type lgssc unregistered [ 1007.092550] LNet: 33601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.102804] LNetError: 33601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.128366] LNet: Removed LNI 192.168.204.117@tcp [ 1007.981430] Key type .llcrypt unregistered [ 1007.983737] Key type ._llcrypt unregistered [ 1034.549435] Key type ._llcrypt registered [ 1034.553170] Key type .llcrypt registered [ 1034.684344] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 1036.043463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1036.082325] alg: No test for adler32 (adler32-zlib) [ 1037.498693] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 1037.828741] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 1039.608485] Key type lgssc registered [ 1041.021973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1050.034963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1050.080452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.658052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1055.724497] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1061.859117] LustreError: 35017:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1061.881694] LustreError: 35017:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1066.471590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.981316] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1067.221805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1072.123621] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1081.649947] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1090.809865] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1100.386715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1101.279333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.389088] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1111.565371] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1116.431670] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1121.831640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1136.769129] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 18:24:37 (1787869477) [ 1138.812769] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1140.689497] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1147.916257] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 18:24:49 (1787869489) [ 1158.551068] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 18:24:59 (1787869499) [ 1159.990948] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1170.525333] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1170.532532] Lustre: Skipped 2 previous similar messages [ 1172.660679] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 18:25:13 (1787869513) [ 1174.981028] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1174.989382] Lustre: Skipped 2 previous similar messages [ 1184.137423] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1184.148305] Lustre: Skipped 4 previous similar messages [ 1200.152925] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 18:25:41 (1787869541) [ 1204.434913] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1211.026629] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1222.270903] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 18:26:03 (1787869563) [ 1224.849112] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1224.854468] Lustre: Skipped 6 previous similar messages [ 1243.172422] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 18:26:24 (1787869584) [ 1265.133071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1265.136771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1265.144104] LustreError: Skipped 1 previous similar message [ 1265.148874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1265.159448] Lustre: Skipped 1 previous similar message [ 1270.246704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1270.270668] Lustre: Skipped 2 previous similar messages [ 1275.360145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1275.362061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1275.373699] Lustre: Skipped 1 previous similar message [ 1275.711163] Lustre: server umount lustre-OST0000 complete [ 1280.484548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1280.490734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1280.497331] Lustre: Skipped 1 previous similar message [ 1280.510852] Lustre: Skipped 1 previous similar message [ 1286.823993] Lustre: server umount lustre-MDT0000 complete [ 1290.723398] LustreError: 35014:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1290.754493] LustreError: 35014:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1291.805370] LustreError: 34998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869634 with bad export cookie 13728413453944336689 [ 1291.812747] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1291.816719] LustreError: 34998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1292.420671] Lustre: server umount lustre-MDT0001 complete [ 1305.730995] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 1309.114979] Key type lgssc unregistered [ 1309.480524] LNet: 42880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1309.492191] LNetError: 42880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1309.525197] LNet: Removed LNI 192.168.204.117@tcp [ 1310.623650] Key type .llcrypt unregistered [ 1310.629498] Key type ._llcrypt unregistered [ 1330.977368] Key type ._llcrypt registered [ 1330.982244] Key type .llcrypt registered [ 1331.102978] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 1352.368600] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 1354.095177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1354.155054] alg: No test for adler32 (adler32-zlib) [ 1355.400922] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 1355.745216] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 1357.512159] Key type lgssc registered [ 1358.652385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1403.816212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1403.873115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1405.237415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1405.279653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1405.420893] Lustre: lustre-MDT0000: new disk, initializing [ 1405.488728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1405.509944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1410.641688] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1421.902148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1422.046320] Lustre: 46037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1422.099280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1422.108210] Lustre: Skipped 1 previous similar message [ 1422.276240] Lustre: lustre-MDT0001: new disk, initializing [ 1422.502697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1422.536348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1422.558081] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1427.931283] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1439.050937] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1446.070823] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1454.345078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1454.715531] Lustre: lustre-OST0000: new disk, initializing [ 1454.721799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1454.733098] Lustre: 48004:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1454.840815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.023511] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1464.880034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1464.886644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1465.066548] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1474.538612] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1480.371520] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1480.576983] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1486.747209] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1487.064383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1490.408196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1490.418653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1490.434551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1490.914984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1490.922662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1490.938978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1496.102581] Lustre: server umount lustre-OST0000 complete [ 1501.156472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.160893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1501.183240] Lustre: Skipped 1 previous similar message [ 1501.212516] Lustre: Skipped 1 previous similar message [ 1506.273192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1506.286276] Lustre: Skipped 1 previous similar message [ 1507.112661] Lustre: server umount lustre-MDT0000 complete [ 1511.693795] LustreError: 46031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787869854 with bad export cookie 11712041063920843022 [ 1511.698653] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1511.705867] LustreError: 46031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1512.164347] Lustre: server umount lustre-MDT0001 complete [ 1524.405200] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 18:31:05 (1787869865) [ 1532.435861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1533.423397] LustreError: 50466:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1533.444354] LustreError: 50466:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1533.566611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.043518] LustreError: 50467:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1539.147399] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1543.139884] LustreError: 50466:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.272651] LustreError: 50467:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1548.944907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.386402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1553.599052] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1562.789321] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1571.454209] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1579.550584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1580.074996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1585.631888] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1593.237898] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1609.703152] LustreError: 53020:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1610.832807] Lustre: Modifying parameter general.timeout=1 in log params [ 1611.952136] Lustre: Modifying parameter general.timeout=2 in log params [ 1614.186916] Lustre: Modifying parameter general.timeout=4 in log params [ 1614.200335] Lustre: Skipped 1 previous similar message [ 1618.617408] Lustre: Modifying parameter general.timeout=9 in log params [ 1618.629365] Lustre: Skipped 4 previous similar messages [ 1627.427790] Lustre: Modifying parameter general.timeout=20 in log params [ 1627.432824] Lustre: Skipped 10 previous similar messages [ 1643.846246] Lustre: Modifying parameter general.timeout=39 in log params [ 1643.850165] Lustre: Skipped 18 previous similar messages [ 1676.165989] Lustre: Modifying parameter general.timeout=72 in log params [ 1676.170960] Lustre: Skipped 32 previous similar messages [ 1748.044662] Lustre: Disabling parameter general.timeout= in log params [ 1748.056776] Lustre: Skipped 57 previous similar messages [ 1752.672480] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 18:34:53 (1787870093) [ 1756.313860] Lustre: 59971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1758.617604] Lustre: 60069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1761.037801] Lustre: 60167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1766.164090] Lustre: 60380:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1766.176775] Lustre: 60380:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1804.600494] Lustre: 61852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1804.623727] Lustre: 61852:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1804.669960] LustreError: 61852:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1810.142658] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 18:35:51 (1787870151) [ 1829.348274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1829.363931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1829.375050] Lustre: Skipped 2 previous similar messages [ 1829.387042] Lustre: Skipped 1 previous similar message [ 1834.001851] Lustre: server umount lustre-MDT0000 complete [ 1834.465656] LustreError: 50467:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1834.496373] LustreError: 50467:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1838.628417] LustreError: 50446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870181 with bad export cookie 11712041063920843911 [ 1838.637693] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1838.650422] LustreError: 50446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1839.160614] Lustre: server umount lustre-MDT0001 complete [ 1855.498408] Lustre: server umount lustre-OST0000 complete [ 1889.519697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1889.903462] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1889.971757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1889.983189] Lustre: Skipped 22 previous similar messages [ 1890.508815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1896.898162] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1907.838891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1907.968722] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1907.998850] Lustre: 64386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1914.246217] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1924.261885] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1931.624973] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1941.066404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1941.339687] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1941.709352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.714800] Lustre: Skipped 1 previous similar message [ 1949.603715] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 1958.872722] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1961.156084] Lustre: 66954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1971.172503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1971.183464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1971.194617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1971.199599] Lustre: Skipped 1 previous similar message [ 1976.274972] Lustre: server umount lustre-OST0000 complete [ 1981.927141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1981.957350] Lustre: Skipped 2 previous similar messages [ 1987.872425] Lustre: server umount lustre-MDT0000 complete [ 1991.146616] LustreError: 64393:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1991.164541] LustreError: 64393:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1992.904286] LustreError: 66247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870335 with bad export cookie 11712041063920847054 [ 1992.911864] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1992.922289] LustreError: 66247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1993.264784] Lustre: server umount lustre-MDT0001 complete [ 2007.720482] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 2010.830723] Key type lgssc unregistered [ 2011.105712] LNet: 68169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2011.109327] LNetError: 68169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2011.141859] LNet: Removed LNI 192.168.204.117@tcp [ 2012.173887] Key type .llcrypt unregistered [ 2012.179766] Key type ._llcrypt unregistered [ 2043.345786] Key type ._llcrypt registered [ 2043.349876] Key type .llcrypt registered [ 2043.470329] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 2045.086618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2045.181758] alg: No test for adler32 (adler32-zlib) [ 2046.300479] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 2046.552496] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 2048.248279] Key type lgssc registered [ 2049.721951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2059.254567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2059.291929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.975358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2066.308723] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2070.502215] LustreError: 69587:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2070.524285] LustreError: 69587:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2075.621918] LustreError: 69588:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2076.953664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.705885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2084.924735] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2094.865034] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2102.485539] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2109.971310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2110.704472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2118.141213] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2127.341063] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2140.599752] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2154.470720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2154.480988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.507741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2156.015499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2156.016987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2156.024872] Lustre: Skipped 1 previous similar message [ 2156.048895] Lustre: Skipped 1 previous similar message [ 2160.274890] Lustre: server umount lustre-MDT0000 complete [ 2161.125591] LustreError: 69588:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2161.137950] LustreError: 69588:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2163.731565] LustreError: 69569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870506 with bad export cookie 3731076265489880579 [ 2163.738309] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2163.752521] LustreError: 69569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2164.169496] Lustre: server umount lustre-MDT0001 complete [ 2179.423395] Lustre: server umount lustre-OST0000 complete [ 2215.655119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2216.165668] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2216.227212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2216.642510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.215890] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2231.961817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2232.043976] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2232.071639] Lustre: 74461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2232.738940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2237.892441] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2246.754100] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2253.363673] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2261.483644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2261.707262] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2262.184831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2269.376552] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2278.137401] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2280.518960] Lustre: 77038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2282.935966] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2290.150660] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2290.165683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2290.187665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2290.195083] Lustre: Skipped 2 previous similar messages [ 2295.278746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2295.300095] Lustre: Skipped 1 previous similar message [ 2295.632434] Lustre: server umount lustre-OST0000 complete [ 2300.389406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.412367] Lustre: Skipped 1 previous similar message [ 2300.420729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2300.434552] Lustre: Skipped 1 previous similar message [ 2306.343974] Lustre: server umount lustre-MDT0000 complete [ 2310.631437] LustreError: 74473:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.661301] LustreError: 74473:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2310.743757] LustreError: 76342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870653 with bad export cookie 3731076265489881426 [ 2310.753657] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2310.757048] LustreError: 76342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2311.258201] Lustre: server umount lustre-MDT0001 complete [ 2324.060464] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 2327.268770] Key type lgssc unregistered [ 2327.563494] LNet: 78256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2327.576339] LNetError: 78256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2327.606741] LNet: Removed LNI 192.168.204.117@tcp [ 2328.463124] Key type .llcrypt unregistered [ 2328.464733] Key type ._llcrypt unregistered [ 2354.541123] Key type ._llcrypt registered [ 2354.543200] Key type .llcrypt registered [ 2354.657648] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 2355.831544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2355.884056] alg: No test for adler32 (adler32-zlib) [ 2356.931317] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 2357.162086] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 2358.842134] Key type lgssc registered [ 2360.211687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2369.096703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2369.114455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2370.738392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2375.642683] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2380.263212] LustreError: 79675:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.299403] LustreError: 79675:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2385.367843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2385.377995] LustreError: 79676:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2385.989591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2391.682291] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2399.716971] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2407.424441] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2415.964100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2416.468240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2425.261956] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2434.882571] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2441.013889] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2442.028449] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2444.140456] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2444.144782] Lustre: Skipped 1 previous similar message [ 2446.287051] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2446.290329] Lustre: Skipped 1 previous similar message [ 2451.049112] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2451.054987] Lustre: Skipped 4 previous similar messages [ 2459.162857] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2459.173962] Lustre: Skipped 7 previous similar messages [ 2476.094244] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2476.103817] Lustre: Skipped 14 previous similar messages [ 2508.523436] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2508.530387] Lustre: Skipped 32 previous similar messages [ 2554.648803] Lustre: 87782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2558.792034] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 18:48:19 (1787870899) [ 2561.189595] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2563.837115] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 18:48:24 (1787870904) [ 2570.213502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2570.214835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2570.219801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2570.226544] Lustre: Skipped 1 previous similar message [ 2574.230284] Lustre: server umount lustre-OST0000 complete [ 2580.449586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2580.454259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2580.466050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2580.484865] Lustre: Skipped 1 previous similar message [ 2580.520356] Lustre: Skipped 2 previous similar messages [ 2585.586598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2585.593677] Lustre: Skipped 1 previous similar message [ 2590.693542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2590.707091] Lustre: Skipped 1 previous similar message [ 2592.224139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2592.319184] Lustre: server umount lustre-MDT0000 complete [ 2595.824653] LustreError: 87992:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2595.863980] LustreError: 87992:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2597.067656] LustreError: 79656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787870939 with bad export cookie 6874626977782534691 [ 2597.072619] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2597.077681] LustreError: 79656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2598.015161] Lustre: server umount lustre-MDT0001 complete [ 2613.066352] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 2615.933252] Key type lgssc unregistered [ 2616.341947] LNet: 89036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2616.353428] LNetError: 89036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2616.377957] LNet: Removed LNI 192.168.204.117@tcp [ 2617.532670] Key type .llcrypt unregistered [ 2617.534574] Key type ._llcrypt unregistered [ 2622.208268] Key type ._llcrypt registered [ 2622.210360] Key type .llcrypt registered [ 2622.306776] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2627.090271] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules [ 2628.377523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2628.432584] alg: No test for adler32 (adler32-zlib) [ 2629.619657] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 2630.116528] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 2631.856220] Key type lgssc registered [ 2633.411132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2635.032733] LustreError: 90256:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2653.421377] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 2656.098290] LustreError: 90256:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2662.251897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2662.284205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2663.807944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.874366] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2674.146393] LustreError: 91256:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.178154] LustreError: 91256:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2679.275248] LustreError: 91255:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.573670] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 18:50:20 (1787871020) [ 2684.387593] LustreError: 91256:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2685.701453] LustreError: 91287:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2685.719803] LustreError: 91287:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2685.730669] LustreError: 91287:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 2686.155778] Lustre: server umount lustre-MDT0000 complete [ 2702.790181] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 2706.390113] Key type lgssc unregistered [ 2706.766831] LNet: 92785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2706.774728] LNetError: 92785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2706.805272] LNet: Removed LNI 192.168.204.117@tcp [ 2707.938249] Key type .llcrypt unregistered [ 2707.940075] Key type ._llcrypt unregistered [ 2730.915879] Key type ._llcrypt registered [ 2730.919157] Key type .llcrypt registered [ 2731.011773] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 2732.715269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2732.767579] alg: No test for adler32 (adler32-zlib) [ 2733.911141] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 2734.187772] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 2735.840200] Key type lgssc registered [ 2737.070503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2745.498451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2745.529286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2747.067444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2753.811403] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2756.587593] LustreError: 94003:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2756.596830] LustreError: 94003:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2761.699733] LustreError: 94004:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2765.433283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.949838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2771.604138] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2780.636846] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2787.803917] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2795.248342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2795.778404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.717916] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2810.644302] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2875.586078] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 18:53:36 (1787871216) [ 2877.795468] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2880.503926] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 18:53:41 (1787871221) [ 2888.680877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2888.690549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2888.712548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2889.206248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2889.211198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2889.232262] Lustre: Skipped 1 previous similar message [ 2889.236228] Lustre: Skipped 1 previous similar message [ 2892.662571] Lustre: server umount lustre-MDT0000 complete [ 2894.305833] LustreError: 94734:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.341512] LustreError: 94734:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2897.229587] LustreError: 93984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871240 with bad export cookie 11373771109750852965 [ 2897.248706] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2897.261826] LustreError: 93984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2897.543434] Lustre: server umount lustre-MDT0001 complete [ 2912.652631] Lustre: server umount lustre-OST0000 complete [ 2924.993989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2925.685452] LustreError: 98320:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2925.908380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2931.098412] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2931.175707] LustreError: 98321:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.391348] LustreError: 98320:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.405347] LustreError: 98320:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2940.718377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2941.517446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2946.704564] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 2955.054127] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2960.855795] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2975.475837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2975.725523] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2975.735743] LustreError: 98307:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2975.784730] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2975.799980] LustreError: 100405:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2975.805197] LustreError: 100405:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2975.812615] LustreError: 100405:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2975.891807] Lustre: server umount lustre-OST0000 complete [ 2975.894410] LustreError: 100405:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2985.188196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2985.430291] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2985.443105] LustreError: 100877:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2985.447282] LustreError: 98308:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2985.492263] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2985.502731] LustreError: 100873:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2985.513584] LustreError: 100873:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2985.522464] LustreError: 100873:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2985.596508] Lustre: server umount lustre-OST0000 complete [ 2985.598597] LustreError: 100873:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2996.313809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2996.527322] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2996.531625] Lustre: Found index 0 for lustre-OST0000, updating log [ 2996.536440] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2996.655619] Lustre: lustre-OST0000: new disk, initializing [ 2996.660364] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2996.667220] Lustre: 101408:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2997.022064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3004.052792] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3008.317236] Lustre: Failing over lustre-OST0000 [ 3008.518788] Lustre: server umount lustre-OST0000 complete [ 3012.578334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3012.601482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3012.609504] Lustre: Skipped 2 previous similar messages [ 3015.653643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3015.673662] Lustre: Skipped 1 previous similar message [ 3018.877719] Lustre: server umount lustre-MDT0000 complete [ 3020.773161] LustreError: 98315:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3020.795042] LustreError: 98315:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3022.927933] LustreError: 98301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871365 with bad export cookie 11373771109750855709 [ 3022.946508] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3023.489837] Lustre: server umount lustre-MDT0001 complete [ 3032.484770] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 18:56:13 (1787871373) [ 3048.618236] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 3061.298541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.008881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3067.187822] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3076.109969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3081.126240] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3084.193324] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3090.328333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3096.487843] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3105.396008] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3105.479779] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3105.573772] Lustre: lustre-OST0001: new disk, initializing [ 3105.578051] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3105.584088] Lustre: 106069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3105.898981] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3105.905907] Lustre: Skipped 2 previous similar messages [ 3112.813528] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3114.648352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3114.667221] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3114.762149] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3121.963904] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3131.780673] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3136.904232] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.17@tcp (stopping) [ 3138.537212] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3138.549636] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3138.562508] Lustre: Skipped 1 previous similar message [ 3142.376883] Lustre: server umount lustre-MDT0001 complete [ 3145.702716] LustreError: 107020:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3145.729631] LustreError: 107020:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 3152.137872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3152.317874] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3152.326515] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3152.414979] Lustre: 103744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3153.676807] LustreError: 107606:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3153.681754] LustreError: 107606:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3153.687799] LustreError: 103746:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3153.748444] LustreError: 107611:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.117@tcp: cfg command failed: rc = -17 [ 3153.758935] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3153.758935] [ 3158.516500] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3158.639276] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3169.672572] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 18:58:30 (1787871510) [ 3186.762301] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 3204.469656] Lustre: 109683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3231.561291] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3236.449605] Lustre: 110819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3326.952760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3326.967864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3326.986748] Lustre: Skipped 2 previous similar messages [ 3327.000760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3327.008819] Lustre: Skipped 5 previous similar messages [ 3330.395742] Lustre: server umount lustre-MDT0000 complete [ 3332.580659] LustreError: 107020:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3332.589638] LustreError: 107020:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 3334.277127] LustreError: 103734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871677 with bad export cookie 11373771109750856703 [ 3334.279352] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3334.283706] LustreError: 103734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3334.501879] Lustre: server umount lustre-MDT0001 complete [ 3348.735136] Lustre: server umount lustre-OST0000 complete [ 3363.714292] Lustre: server umount lustre-OST0001 complete [ 3370.142810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3381.207899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3391.425556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3403.656287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3426.112763] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 3438.803382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3438.828588] Lustre: lustre-MDT0000: reset Object Index mappings [ 3439.570587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3439.584590] Lustre: Skipped 1 previous similar message [ 3444.616981] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3455.373602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.389994] Lustre: lustre-MDT0001: reset Object Index mappings [ 3462.195174] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3465.820217] Lustre: 115936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3472.880603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3480.824722] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3480.974957] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3484.927932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3490.345387] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3496.478665] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3496.484681] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3498.974454] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3508.587400] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3513.481705] Lustre: 117812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3571.015168] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 19:05:12 (1787871912) [ 3598.826924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3598.829440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3598.842191] Lustre: Skipped 5 previous similar messages [ 3598.861419] Lustre: Skipped 7 previous similar messages [ 3601.980613] Lustre: server umount lustre-MDT0000 complete [ 3603.944207] LustreError: 114786:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3603.968607] LustreError: 114786:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 12 previous similar messages [ 3605.762603] LustreError: 117833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787871948 with bad export cookie 11373771109750903554 [ 3605.764921] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3605.775645] LustreError: 117833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3620.323312] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3620.807813] Lustre: server umount lustre-MDT0001 complete [ 3631.793274] Lustre: server umount lustre-OST0000 complete [ 3642.787901] Lustre: server umount lustre-OST0001 complete [ 3649.939253] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 3661.070725] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 3704.162137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.481755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3704.518110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3704.657604] Lustre: lustre-MDT0000: new disk, initializing [ 3704.791598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3704.801677] Lustre: Skipped 3 previous similar messages [ 3704.815683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3710.865695] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3719.807887] Lustre: server umount lustre-MDT0000 complete [ 3731.782959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.264986] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3732.278854] Lustre: 122086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3732.299602] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3738.177876] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3744.634854] Lustre: server umount lustre-MDT0000 complete [ 3756.654043] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 19:08:17 (1787872097) [ 3758.549920] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3760.971767] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 19:08:21 (1787872101) [ 3781.869822] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 3794.116579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3799.824762] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3809.256339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.457766] Lustre: 124137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3809.568371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3809.572955] Lustre: Skipped 1 previous similar message [ 3809.709398] Lustre: lustre-MDT0001: new disk, initializing [ 3809.895830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3809.912267] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3814.992778] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3820.294836] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3827.890699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3828.237193] Lustre: lustre-OST0000: new disk, initializing [ 3828.241443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3828.250300] Lustre: 125715:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3829.616776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3829.657693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3829.788776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3834.909719] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3846.113777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3846.392857] Lustre: lustre-OST0001: new disk, initializing [ 3846.401939] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3846.419608] Lustre: 126584:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3848.410422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3848.428976] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3848.513808] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3854.383747] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 3865.047222] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3870.235931] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4287.298979] Lustre: 133009:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4287.310323] Lustre: 133009:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4287.318054] Lustre: 133009:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4287.333044] Lustre: 133009:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4287.339438] Lustre: 133009:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4287.344488] Lustre: 133009:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4634.100878] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 19:22:54 (1787872974) [ 4654.573813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4654.592646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4654.611466] Lustre: Skipped 3 previous similar messages [ 4654.620934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4654.628255] Lustre: Skipped 8 previous similar messages [ 4657.190414] Lustre: server umount lustre-MDT0000 complete [ 4660.712316] LustreError: 127398:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4660.731035] LustreError: 127398:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 4661.548288] LustreError: 133021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787873004 with bad export cookie 11373771109750972931 [ 4661.553084] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4661.565471] LustreError: 133021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4665.826552] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4665.839359] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4665.850018] Lustre: Skipped 4 previous similar messages [ 4665.863752] Lustre: Skipped 5 previous similar messages [ 4668.479820] Lustre: server umount lustre-MDT0001 complete [ 4679.496752] Lustre: server umount lustre-OST0000 complete [ 4690.260697] Lustre: server umount lustre-OST0001 complete [ 4698.518797] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 4712.007698] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 4758.867142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4759.150257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4759.217551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4759.316393] Lustre: lustre-MDT0000: new disk, initializing [ 4759.466330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4759.477329] Lustre: Skipped 5 previous similar messages [ 4759.493413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4764.811717] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4779.200341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4779.392925] Lustre: 136077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4779.448974] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4779.452821] Lustre: Skipped 1 previous similar message [ 4779.522724] Lustre: lustre-MDT0001: new disk, initializing [ 4779.776757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4779.789286] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4785.751819] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4799.293335] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4807.478622] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4818.218083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4818.769532] Lustre: lustre-OST0000: new disk, initializing [ 4818.773139] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4818.781355] Lustre: 138043:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4818.889054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4818.906233] Lustre: Skipped 1 previous similar message [ 4820.610120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4820.620995] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4820.737387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4827.834566] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 4840.082799] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4853.589497] Lustre: lustre-MDD0000: changelog on [ 4857.851519] Lustre: lustre-MDD0001: changelog on [ 6952.307440] Lustre: DEBUG MARKER: conf-sanity test_135: @@@@@@ FAIL: creation thread is running. Is changelog reader stuck? [ 6973.041667] Lustre: lustre-MDD0001: changelog off [ 6978.040828] Lustre: lustre-MDD0000: changelog off [ 7018.996688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7018.997878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7019.000194] LustreError: Skipped 1 previous similar message [ 7019.032691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7019.046511] Lustre: Skipped 2 previous similar messages [ 7020.513613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7020.514458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7025.635628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7025.638285] Lustre: Skipped 2 previous similar messages [ 7030.753647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7030.756454] Lustre: Skipped 1 previous similar message [ 7033.312349] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7033.498909] Lustre: server umount lustre-OST0000 complete [ 7037.935828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7037.949189] LustreError: Skipped 6 previous similar messages [ 7037.957030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7040.994197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7040.995340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7041.036686] Lustre: Skipped 2 previous similar messages [ 7043.347887] Lustre: server umount lustre-MDT0000 complete [ 7046.115801] LustreError: 136086:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7046.141788] LustreError: 136086:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 7048.765884] LustreError: 136068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875391 with bad export cookie 11373771109751792008 [ 7048.772083] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7048.790439] LustreError: 136068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7049.210217] Lustre: server umount lustre-MDT0001 complete [ 7062.182518] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 7065.099724] Key type lgssc unregistered [ 7065.351158] LNet: 146874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7065.358872] LNetError: 146874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7066.411077] LNet: Removed LNI 192.168.204.117@tcp [ 7067.361193] Key type .llcrypt unregistered [ 7067.367765] Key type ._llcrypt unregistered [ 7088.286731] Key type ._llcrypt registered [ 7088.289833] Key type .llcrypt registered [ 7088.370298] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 7104.508777] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 7105.967749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7106.056725] alg: No test for adler32 (adler32-zlib) [ 7107.245385] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 7107.605111] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 7109.280220] Key type lgssc registered [ 7110.555364] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7151.454364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7151.496082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7152.823441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7152.859229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7152.970910] Lustre: lustre-MDT0000: new disk, initializing [ 7153.074380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7153.104342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7158.773062] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7169.262327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7169.365578] Lustre: 150078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7169.445409] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7169.452906] Lustre: Skipped 1 previous similar message [ 7169.594467] Lustre: lustre-MDT0001: new disk, initializing [ 7169.726215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7169.781669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7169.788901] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7173.805517] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7185.725416] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7192.454129] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7201.004834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7201.376304] Lustre: lustre-OST0000: new disk, initializing [ 7201.385264] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7201.401100] Lustre: 152048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7201.552390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7208.328404] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7211.107569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7211.121160] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7211.243582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7219.773123] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7226.802220] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_module obdecho/obdecho [ 7229.043906] LustreError: 153052:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7229.053733] LustreError: 153052:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7231.117281] LustreError: 153148:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7246.821744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7246.828712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7246.840465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7247.334492] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7247.354465] Lustre: Skipped 1 previous similar message [ 7247.356840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7247.373800] Lustre: Skipped 2 previous similar messages [ 7250.708293] Lustre: server umount lustre-MDT0000 complete [ 7254.802445] LustreError: 150070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875597 with bad export cookie 6443983782408364162 [ 7254.807609] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7254.817599] LustreError: 150070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7255.301282] Lustre: server umount lustre-MDT0001 complete [ 7269.869640] Lustre: server umount lustre-OST0000 complete [ 7291.619459] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 7301.492864] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 7335.253037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7335.420292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7335.467503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7335.561304] Lustre: lustre-MDT0000: new disk, initializing [ 7335.632572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7335.649113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7339.401489] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7347.734632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7347.865859] Lustre: 156867:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7347.921194] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7347.925089] Lustre: Skipped 1 previous similar message [ 7347.991403] Lustre: lustre-MDT0001: new disk, initializing [ 7348.081079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7348.130116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7348.141968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7352.715643] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7361.872474] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7367.853517] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7375.575575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7376.067929] Lustre: lustre-OST0000: new disk, initializing [ 7376.071046] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7376.077291] Lustre: 158834:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7376.220984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7382.938292] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7384.635692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7384.642619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7384.751706] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7391.838865] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7395.584536] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7395.889455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7399.375586] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7399.555685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7405.550020] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7405.551800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7405.562895] LustreError: Skipped 1 previous similar message [ 7405.564913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7405.573406] Lustre: Skipped 1 previous similar message [ 7405.596153] Lustre: Skipped 1 previous similar message [ 7407.285680] Lustre: server umount lustre-OST0000 complete [ 7414.756291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7414.764425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7414.785127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7417.670776] Lustre: server umount lustre-MDT0000 complete [ 7420.909708] LustreError: 156874:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7420.930391] LustreError: 156874:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 7421.396514] LustreError: 156857:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875764 with bad export cookie 6443983782408365114 [ 7421.414533] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7421.883439] Lustre: server umount lustre-MDT0001 complete [ 7425.933685] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 20:09:26 (1787875766) [ 7427.339687] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7429.108425] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 20:09:30 (1787875770) [ 7436.833248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7437.332687] LustreError: 161250:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7437.412224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7442.402521] LustreError: 161251:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7442.908079] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7447.535411] LustreError: 161250:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7451.903902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7456.157490] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7462.159766] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7467.008540] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7472.475935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7472.684153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7472.690730] Lustre: Skipped 1 previous similar message [ 7477.245396] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7482.447954] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7487.981647] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7487.990091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7487.997139] Lustre: Skipped 3 previous similar messages [ 7488.005934] Lustre: Skipped 4 previous similar messages [ 7489.755594] Lustre: server umount lustre-MDT0000 complete [ 7492.841088] LustreError: 161986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787875835 with bad export cookie 6443983782408366003 [ 7492.842986] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7492.849714] LustreError: 161986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7493.089924] LustreError: 161246:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7497.122270] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7497.127208] Lustre: Skipped 3 previous similar messages [ 7499.164645] Lustre: server umount lustre-MDT0001 complete [ 7504.647955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7515.832037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7516.046201] LustreError: 165090:0:(llog.c:1655:llog_backup()) MGC192.168.204.117@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7516.055122] Lustre: 165090:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.117@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7522.788820] LustreError: 165095:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7522.799531] LustreError: 165095:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 7539.552836] LustreError: 165097:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7539.573162] LustreError: 165097:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 14 previous similar messages [ 7573.473613] LustreError: 165096:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7573.504274] LustreError: 165096:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 25 previous similar messages [ 7577.060169] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7577.071836] Lustre: 165090:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.117@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7618.053253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7623.154649] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7624.371721] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7636.829990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7638.504031] LustreError: 163082:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7638.536683] LustreError: 163082:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 43 previous similar messages [ 7643.950622] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7644.157416] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7652.397501] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7653.835711] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7662.003025] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7664.708517] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7700.450818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7700.451452] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7700.456719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7700.456732] Lustre: Skipped 1 previous similar message [ 7700.490777] Lustre: Skipped 3 previous similar messages [ 7701.457771] Lustre: server umount lustre-MDT0000 complete [ 7706.103065] LustreError: 161230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876048 with bad export cookie 6443983782408372457 [ 7706.106390] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7706.114510] LustreError: 161230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7706.503377] Lustre: server umount lustre-MDT0001 complete [ 7721.050733] Lustre: server umount lustre-OST0000 complete [ 7754.107739] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 7764.595850] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 7803.867433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7804.060952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7804.085942] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7804.153279] Lustre: lustre-MDT0000: new disk, initializing [ 7804.214051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7804.217552] Lustre: Skipped 1 previous similar message [ 7804.227246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7808.693748] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7819.818854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7819.897365] Lustre: 170205:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7820.107864] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7823.831639] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7833.984556] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7839.983752] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7848.600752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7848.975895] Lustre: lustre-OST0000: new disk, initializing [ 7848.979264] Lustre: Skipped 1 previous similar message [ 7848.995716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7849.003160] Lustre: Skipped 2 previous similar messages [ 7849.031888] Lustre: 172173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7855.947276] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7856.168025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7856.173619] Lustre: Skipped 1 previous similar message [ 7856.176659] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7856.267358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7865.927477] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7871.150808] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7871.316868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7876.541089] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7876.710074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7881.699363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7881.707657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7881.725984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7881.733163] Lustre: Skipped 2 previous similar messages [ 7884.837294] Lustre: server umount lustre-OST0000 complete [ 7891.939382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7891.950913] LustreError: Skipped 1 previous similar message [ 7894.758033] Lustre: server umount lustre-MDT0000 complete [ 7897.569187] LustreError: 171014:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7897.612123] LustreError: 171014:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 7899.376937] LustreError: 170198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876242 with bad export cookie 6443983782408389292 [ 7899.392822] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7899.395966] LustreError: 170198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7899.845455] Lustre: server umount lustre-MDT0001 complete [ 7911.460675] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 20:17:32 (1787876252) [ 7919.598364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7925.465593] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7936.604249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7943.233816] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7951.682641] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7957.757472] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7965.409903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7973.705863] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 7981.470445] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7991.670275] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 20:18:52 (1787876332) [ 7996.902707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7996.904191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7996.919750] Lustre: Skipped 3 previous similar messages [ 7996.923450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7996.923459] Lustre: Skipped 5 previous similar messages [ 7996.952095] LustreError: Skipped 1 previous similar message [ 8002.069265] Lustre: server umount lustre-OST0000 complete [ 8012.284765] Lustre: server umount lustre-MDT0000 complete [ 8016.591027] LustreError: 174567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876359 with bad export cookie 6443983782408390174 [ 8016.599983] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8016.609272] LustreError: 174567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8017.065533] Lustre: server umount lustre-MDT0001 complete [ 8029.448108] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 8032.589438] Key type lgssc unregistered [ 8032.891163] LNet: 178321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8032.899197] LNetError: 178321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8032.912988] LNet: Removed LNI 192.168.204.117@tcp [ 8033.899093] Key type .llcrypt unregistered [ 8033.901870] Key type ._llcrypt unregistered [ 8052.362515] Key type ._llcrypt registered [ 8052.365136] Key type .llcrypt registered [ 8052.474749] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 8053.181673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8053.253491] alg: No test for adler32 (adler32-zlib) [ 8054.262503] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 8054.392180] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 8056.090977] Key type lgssc registered [ 8057.065669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8064.699880] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8064.732046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8080.352488] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8085.472757] LustreError: 179564:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.117@tcp: failed processing log, type 4: rc = -110 [ 8090.592592] LustreError: 179564:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8090.611980] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8090.621432] LustreError: MGC192.168.204.117@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8090.635399] LustreError: 179564:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8090.657328] LustreError: 179564:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8090.673902] LustreError: 179564:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8090.743171] Lustre: server umount lustre-OST0000 complete [ 8090.747173] LustreError: 179564:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8101.910631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8103.257024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8107.238585] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8113.635154] LustreError: 180065:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8113.657950] LustreError: 180065:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8116.418143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8117.167536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8122.869249] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8131.599154] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8139.074118] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8150.614534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8150.933791] LustreError: 181882:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8150.951236] Lustre: 181882:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.117@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8151.157713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8158.993823] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8167.503449] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8172.019152] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8172.031240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8172.036114] Lustre: Skipped 2 previous similar messages [ 8172.056080] Lustre: Skipped 2 previous similar messages [ 8176.432466] Lustre: server umount lustre-MDT0000 complete [ 8177.125038] LustreError: 180064:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8177.143552] LustreError: 180064:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8180.410325] LustreError: 180045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876523 with bad export cookie 3553099483378352248 [ 8180.422373] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8180.595290] Lustre: server umount lustre-MDT0001 complete [ 8195.237272] Lustre: server umount lustre-OST0000 complete [ 8212.822960] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 8224.602646] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 8271.898207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8272.289583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8272.344471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8272.464091] Lustre: lustre-MDT0000: new disk, initializing [ 8272.595034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8272.607787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8278.523377] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8292.938498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8293.094599] Lustre: 185413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8293.163956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8293.168799] Lustre: Skipped 1 previous similar message [ 8293.246341] Lustre: lustre-MDT0001: new disk, initializing [ 8293.381078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8293.450371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8293.457832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8299.818326] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8310.928447] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8317.813804] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8327.066057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8327.608918] Lustre: lustre-OST0000: new disk, initializing [ 8327.614664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8327.621564] Lustre: 187384:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8327.729995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8333.939697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8333.957327] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8334.124080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8335.710968] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8347.045577] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8351.790032] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8352.225749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8357.349505] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8357.608893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8359.908375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8359.911512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8359.919571] LustreError: Skipped 1 previous similar message [ 8359.940188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8359.950434] Lustre: Skipped 1 previous similar message [ 8365.026168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8365.032025] Lustre: Skipped 1 previous similar message [ 8370.158454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8370.181000] Lustre: Skipped 1 previous similar message [ 8374.240143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8374.513331] Lustre: server umount lustre-OST0000 complete [ 8380.387129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8380.395371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8380.412124] Lustre: Skipped 1 previous similar message [ 8380.419242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8380.426581] Lustre: Skipped 1 previous similar message [ 8386.182502] Lustre: server umount lustre-MDT0000 complete [ 8390.499380] LustreError: 186209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876733 with bad export cookie 3553099483378352962 [ 8390.506726] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8390.523334] LustreError: 186209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8391.139440] Lustre: server umount lustre-MDT0001 complete [ 8403.903440] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 20:25:44 (1787876744) [ 8423.191100] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 8426.515927] Key type lgssc unregistered [ 8426.789979] LNet: 190334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8426.802392] LNetError: 190334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8426.827249] LNet: Removed LNI 192.168.204.117@tcp [ 8427.637351] Key type .llcrypt unregistered [ 8427.641951] Key type ._llcrypt unregistered [ 8445.783227] Key type ._llcrypt registered [ 8445.785608] Key type .llcrypt registered [ 8445.900713] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 8446.916353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8447.073756] alg: No test for adler32 (adler32-zlib) [ 8448.296396] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 8448.615139] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 8450.392174] Key type lgssc registered [ 8451.536411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8460.471841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8460.489412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8461.964728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8467.498825] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8471.524920] LustreError: 191553:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8471.542291] LustreError: 191553:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8476.642807] LustreError: 191554:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8477.604812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8478.104578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8483.549121] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8491.172640] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8497.143452] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8507.482532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8508.011941] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8508.021226] LustreError: 193419:0:(llog.c:1696:llog_backup()) MGC192.168.204.117@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8508.039872] Lustre: 193419:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.117@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8508.259609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8516.873936] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8526.691935] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8532.964862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8532.976286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8532.991454] Lustre: Skipped 2 previous similar messages [ 8533.019456] Lustre: Skipped 2 previous similar messages [ 8538.086136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8538.104684] Lustre: Skipped 2 previous similar messages [ 8543.216283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8543.228814] Lustre: Skipped 2 previous similar messages [ 8544.228335] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8544.364497] Lustre: server umount lustre-MDT0000 complete [ 8548.331104] LustreError: 192498:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8548.369397] LustreError: 192498:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8549.239232] LustreError: 191533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787876892 with bad export cookie 5774321338602628236 [ 8549.247894] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8549.250900] LustreError: 191533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8549.531040] Lustre: server umount lustre-MDT0001 complete [ 8564.622433] Lustre: server umount lustre-OST0000 complete [ 8578.492649] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 8590.721433] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 8637.376134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8637.947383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8637.997758] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8638.092400] Lustre: lustre-MDT0000: new disk, initializing [ 8638.178652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8638.194959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8644.457562] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8657.056948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8657.394799] Lustre: 196947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8657.480566] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8657.482831] Lustre: Skipped 1 previous similar message [ 8657.486407] LustreError: 197731:0:(llog.c:1696:llog_backup()) MGC192.168.204.117@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8657.489885] LustreError: 197731:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8657.492267] Lustre: 197731:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.117@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8657.504411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8657.510944] Lustre: Skipped 1 previous similar message [ 8657.563590] Lustre: lustre-MDT0001: new disk, initializing [ 8657.642799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8657.692953] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8657.709846] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8662.058648] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8673.027247] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8680.560622] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8688.305206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8688.783447] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8688.789535] Lustre: Skipped 1 previous similar message [ 8688.791919] LustreError: 198894:0:(llog.c:1696:llog_backup()) MGC192.168.204.117@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8688.801995] LustreError: 198894:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8688.818973] Lustre: 198894:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.117@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8688.837359] Lustre: 198894:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8688.862650] Lustre: lustre-OST0000: new disk, initializing [ 8688.867730] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8688.874286] Lustre: 198918:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8689.051752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8694.793892] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8698.038393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8698.061134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8698.340405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8706.192689] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8714.559576] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8715.014183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8722.396435] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8722.774636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8729.057510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8729.058903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8729.066036] LustreError: Skipped 1 previous similar message [ 8729.074968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8729.098084] Lustre: Skipped 1 previous similar message [ 8729.134745] Lustre: Skipped 1 previous similar message [ 8733.022931] Lustre: server umount lustre-OST0000 complete [ 8739.299058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8739.315249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8744.549560] Lustre: server umount lustre-MDT0000 complete [ 8749.539800] LustreError: 196959:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8749.619470] LustreError: 196959:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8752.370044] LustreError: 196941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877095 with bad export cookie 5774321338602628950 [ 8752.414645] LustreError: 196941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8752.415844] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8753.420927] Lustre: server umount lustre-MDT0001 complete [ 8770.471182] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 20:31:50 (1787877110) [ 8793.754092] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 8804.764448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8805.124536] LustreError: 202048:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8805.181826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8809.129777] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8815.587643] LustreError: 202049:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8815.622038] LustreError: 202049:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8818.805680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8824.171434] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8829.140706] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8840.809678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8841.577629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8841.596445] Lustre: Skipped 1 previous similar message [ 8851.955784] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8862.133577] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8862.399714] Lustre: lustre-OST0001: new disk, initializing [ 8862.406782] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8862.426659] Lustre: 204360:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8868.983111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8869.000931] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8869.175197] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8870.964489] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8882.835985] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8891.053644] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8893.634716] Lustre: DEBUG MARKER: ADD OST3 [ 8899.715160] loop: module loaded [ 8899.816602] loop0: detected capacity change from 0 to 204800000 [ 8900.176287] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8900.195552] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8900.211453] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8908.098480] loop0: detected capacity change from 0 to 204800000 [ 8908.180498] Lustre: DEBUG MARKER: STOP OST3 [ 8914.181569] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8914.457184] Lustre: lustre-OST0003: new disk, initializing [ 8914.462757] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8914.469468] Lustre: 206414:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8914.591771] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8914.603130] Lustre: Skipped 1 previous similar message [ 8915.934681] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8916.170054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8916.182169] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8916.226898] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8917.126537] Lustre: Failing over lustre-OST0003 [ 8917.289483] Lustre: server umount lustre-OST0003 complete [ 8917.985872] LustreError: 206549:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8921.078781] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8921.093740] Lustre: Skipped 1 previous similar message [ 8921.101239] LustreError: 204060:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8921.233355] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8929.127441] Lustre: DEBUG MARKER: START OST3 again [ 8936.952633] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8937.417941] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8938.598255] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8938.857773] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8938.858599] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8944.582887] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 8966.170224] Lustre: Failing over lustre-OST0003 [ 8966.275977] Lustre: server umount lustre-OST0003 complete [ 8968.177360] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8968.196994] Lustre: Skipped 1 previous similar message [ 8968.203443] LustreError: 203536:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8968.229046] LustreError: 203536:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 12 previous similar messages [ 8988.134117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8988.150341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8988.169028] Lustre: Skipped 1 previous similar message [ 8988.186572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8988.198658] Lustre: Skipped 3 previous similar messages [ 8991.680548] Lustre: server umount lustre-MDT0000 complete [ 8996.428291] LustreError: 205127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877339 with bad export cookie 5774321338602629832 [ 8996.442377] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8996.462425] LustreError: 205127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9003.023967] Lustre: server umount lustre-MDT0001 complete [ 9007.937487] Lustre: server umount lustre-OST0000 complete [ 9012.767359] Lustre: server umount lustre-OST0001 complete [ 9020.755377] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 9029.782250] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 9077.707027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9078.258723] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9078.332833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9078.484357] Lustre: lustre-MDT0000: new disk, initializing [ 9078.689596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9078.701510] Lustre: Skipped 1 previous similar message [ 9078.736370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9085.928925] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 9098.751788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9099.072723] Lustre: 211820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9099.169264] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9099.171824] Lustre: Skipped 1 previous similar message [ 9099.354302] Lustre: lustre-MDT0001: new disk, initializing [ 9099.578691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9099.608033] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9104.795838] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 9116.443981] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9123.870348] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9131.853629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9132.330592] Lustre: lustre-OST0000: new disk, initializing [ 9132.343179] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9132.354230] Lustre: 213787:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9134.268107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9134.284330] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9134.468038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9140.446396] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 9152.468105] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9157.848181] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9158.102154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9163.026813] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9163.293728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9170.406675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9170.421496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9170.433898] LustreError: Skipped 1 previous similar message [ 9170.436902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9170.452722] Lustre: Skipped 6 previous similar messages [ 9170.470586] Lustre: Skipped 8 previous similar messages [ 9172.912910] Lustre: server umount lustre-OST0000 complete [ 9186.302543] Lustre: server umount lustre-MDT0000 complete [ 9190.888672] LustreError: 211830:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9190.902646] LustreError: 211830:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 17 previous similar messages [ 9190.985727] LustreError: 211810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877533 with bad export cookie 5774321338602632128 [ 9190.987855] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9190.990047] LustreError: 211810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9191.375788] Lustre: server umount lustre-MDT0001 complete [ 9194.732821] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 20:38:56 (1787877536) [ 9209.841440] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 9219.961487] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 9263.814887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9264.034818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9264.054700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9264.128836] Lustre: lustre-MDT0000: new disk, initializing [ 9264.214919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9269.755656] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 9280.978290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9281.149340] Lustre: 218401:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9281.538165] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9287.167181] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 9297.885993] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9306.138738] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9314.473321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9314.740161] Lustre: 220373:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9316.766409] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9316.908966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9321.125104] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 9331.791608] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9337.329101] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9337.743623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9343.859154] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9344.117115] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9347.556332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9347.559523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9347.579528] LustreError: Skipped 1 previous similar message [ 9347.582829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9347.619559] Lustre: Skipped 3 previous similar messages [ 9347.643786] Lustre: Skipped 5 previous similar messages [ 9360.864152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9361.234807] Lustre: server umount lustre-OST0000 complete [ 9372.026423] Lustre: server umount lustre-MDT0000 complete [ 9373.155207] LustreError: 218414:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9373.181491] LustreError: 218414:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9376.330334] LustreError: 218396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877719 with bad export cookie 5774321338602633010 [ 9376.340937] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9376.346844] LustreError: 218396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9376.617997] Lustre: server umount lustre-MDT0001 complete [ 9388.007239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9388.852893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9388.857809] Lustre: Skipped 5 previous similar messages [ 9395.056735] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 9405.706562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9411.197522] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 9423.774424] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9434.016349] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9442.484876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9450.451444] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 9459.938747] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9471.459229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9471.468086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9471.494112] Lustre: Skipped 2 previous similar messages [ 9471.498465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9471.503185] Lustre: Skipped 6 previous similar messages [ 9476.580368] Lustre: server umount lustre-OST0000 complete [ 9482.722169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9482.728730] LustreError: Skipped 1 previous similar message [ 9487.545279] Lustre: server umount lustre-MDT0000 complete [ 9492.623808] LustreError: 222571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787877835 with bad export cookie 5774321338602633892 [ 9492.631302] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9492.638336] LustreError: 222571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9492.977559] Lustre: server umount lustre-MDT0001 complete [ 9600.869462] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [ 9604.039836] Key type lgssc unregistered [ 9604.391908] LNet: 226259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9604.398814] LNetError: 226259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9604.412625] LNet: Removed LNI 192.168.204.117@tcp [ 9605.304161] Key type .llcrypt unregistered [ 9605.306106] Key type ._llcrypt unregistered [ 9634.029724] Key type ._llcrypt registered [ 9634.032239] Key type .llcrypt registered [ 9634.164967] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [ 9651.944181] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 9654.085723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9654.122714] alg: No test for adler32 (adler32-zlib) [ 9655.739825] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [ 9656.268121] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [ 9658.088168] Key type lgssc registered [ 9660.153665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9707.326472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9707.354354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9708.682354] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9708.735347] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9708.818078] Lustre: lustre-MDT0000: new disk, initializing [ 9708.987938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9709.039819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9714.426678] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 9727.146450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9727.353909] Lustre: 229622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9727.443071] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9727.447021] Lustre: Skipped 1 previous similar message [ 9727.563637] Lustre: lustre-MDT0001: new disk, initializing [ 9727.713597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9727.791455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9727.807323] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9734.772857] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 9746.206260] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9753.755807] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9763.654459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9764.009141] Lustre: lustre-OST0000: new disk, initializing [ 9764.011746] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9764.020288] Lustre: 231591:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9764.122256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9771.922202] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 9772.668413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9772.684248] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9772.863066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9784.518492] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9791.106462] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9791.391938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9800.041212] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9800.479473] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9808.866087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9808.888948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9808.911481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9808.933445] Lustre: Skipped 1 previous similar message [ 9808.962599] Lustre: Skipped 1 previous similar message [ 9810.355293] Lustre: server umount lustre-OST0000 complete [ 9819.126049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9819.130607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9819.152671] Lustre: Skipped 1 previous similar message [ 9819.167076] Lustre: Skipped 1 previous similar message [ 9822.642678] Lustre: server umount lustre-MDT0000 complete [ 9827.032224] LustreError: 231590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878169 with bad export cookie 9543662275348214481 [ 9827.050552] LustreError: 231590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9827.053231] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9827.385232] Lustre: server umount lustre-MDT0001 complete [ 9856.506655] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [ 9871.509368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9871.921452] LustreError: 234549:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9871.939983] LustreError: 234549:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 9872.026846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9876.341585] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 9877.474183] LustreError: 234550:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9877.495611] LustreError: 234550:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9881.570862] LustreError: 234549:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9886.690982] LustreError: 234550:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9887.197674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9887.698591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9893.738314] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 9897.951218] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9909.093993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9909.657452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9919.866428] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 9931.678049] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9932.096975] Lustre: lustre-OST0001: new disk, initializing [ 9932.104110] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9932.115650] Lustre: 236859:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9932.327089] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9937.649092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9937.675390] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9937.802996] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9940.706589] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [ 9954.019488] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9963.137613] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9977.139791] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 20:51:58 (1787878318) [ 9979.873407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9979.885630] LustreError: Skipped 1 previous similar message [ 9979.894847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9979.920850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9983.991618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9983.994068] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9984.006253] Lustre: Skipped 2 previous similar messages [ 9984.046668] Lustre: Skipped 2 previous similar messages [ 9989.093678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9989.104165] Lustre: Skipped 4 previous similar messages [ 9994.209544] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9994.559879] Lustre: server umount lustre-MDT0000 complete [ 9998.883613] LustreError: 235685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878341 with bad export cookie 9543662275348215363 [ 9998.884199] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9998.892646] LustreError: 235685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9999.295737] Lustre: server umount lustre-MDT0001 complete [10013.833974] Lustre: server umount lustre-OST0000 complete [10015.712578] Lustre: 227844:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787878342/real 1787878342] req@ffff8ca4cf1ad2c0 x1874725960804224/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787878358 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10015.734103] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10018.093920] Lustre: server umount lustre-OST0001 complete [10026.389724] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [10039.795884] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [10080.349268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10080.689580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10080.744533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10080.918351] Lustre: lustre-MDT0000: new disk, initializing [10081.053272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10081.084495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10086.474930] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [10097.836775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10097.971150] Lustre: 240811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10098.058245] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10098.064119] Lustre: Skipped 1 previous similar message [10098.168253] Lustre: lustre-MDT0001: new disk, initializing [10098.381010] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10098.392429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10103.462928] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [10114.284503] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10120.901763] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10128.134679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10128.391459] Lustre: lustre-OST0000: new disk, initializing [10128.395217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10128.401591] Lustre: 242777:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10129.873608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10129.893174] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10130.036333] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10136.248062] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [10150.268943] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10156.336059] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10156.558371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10161.583867] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10161.857667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10165.739900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10165.747621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10165.755605] LustreError: Skipped 1 previous similar message [10165.758978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10165.758986] Lustre: Skipped 4 previous similar messages [10165.789303] Lustre: Skipped 1 previous similar message [10170.251476] Lustre: server umount lustre-OST0000 complete [10185.195323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10185.209629] Lustre: Skipped 6 previous similar messages [10190.305416] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10190.562806] Lustre: server umount lustre-MDT0000 complete [10194.991429] LustreError: 240802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878537 with bad export cookie 9543662275348217498 [10194.997903] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10195.009152] LustreError: 240802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10195.514546] Lustre: server umount lustre-MDT0001 complete [10206.205126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10206.866292] LustreError: 244996:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10207.002625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10207.007583] Lustre: Skipped 2 previous similar messages [10213.054922] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [10216.421338] LustreError: 244997:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10216.452945] LustreError: 244997:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [10225.196409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10230.778137] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [10239.013804] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10247.126484] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10256.656468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10265.650602] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [10274.506950] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10424.297242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10424.312280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10424.320826] Lustre: Skipped 2 previous similar messages [10424.324089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10424.329478] Lustre: Skipped 2 previous similar messages [10436.064714] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10436.335961] Lustre: server umount lustre-OST0000 complete [10446.308053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10446.322562] LustreError: Skipped 1 previous similar message [10448.080704] Lustre: server umount lustre-MDT0000 complete [10451.946402] LustreError: 245725:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10451.970758] LustreError: 245725:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10452.419816] LustreError: 244978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787878795 with bad export cookie 9543662275348218380 [10452.436854] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10452.444871] LustreError: 244978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10453.095513] Lustre: server umount lustre-MDT0001 complete [10540.366064] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [10542.726731] Key type lgssc unregistered [10542.977914] LNet: 248618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10542.992624] LNetError: 248618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10543.009179] LNet: Removed LNI 192.168.204.117@tcp [10543.983452] Key type .llcrypt unregistered [10543.987790] Key type ._llcrypt unregistered [10575.147269] Key type ._llcrypt registered [10575.150656] Key type .llcrypt registered [10575.376136] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [10594.925570] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [10596.365229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10596.375023] alg: No test for adler32 (adler32-zlib) [10597.468853] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [10597.645029] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [10599.298889] Key type lgssc registered [10600.660124] Lustre: Echo OBD driver; http://www.lustre.org/ [10649.555666] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [10665.268537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10665.306726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10666.612600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10666.655171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10666.747818] Lustre: lustre-MDT0000: new disk, initializing [10666.849128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10666.863239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10672.050392] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [10683.123319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10683.226595] Lustre: 252696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10683.294700] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10683.299893] Lustre: Skipped 1 previous similar message [10683.400656] Lustre: lustre-MDT0001: new disk, initializing [10683.510688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10683.561927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10683.579205] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10688.375860] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [10693.386604] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10700.789620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10701.115965] Lustre: lustre-OST0000: new disk, initializing [10701.123226] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10701.128606] Lustre: 254328:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10701.256103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10703.665850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10703.673324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10703.772312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10707.754792] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [10718.785393] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10718.959461] Lustre: lustre-OST0001: new disk, initializing [10718.964465] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10718.972444] Lustre: 255202:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10719.084859] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10725.122326] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [10725.974371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10725.984084] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10726.093024] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10737.247354] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10743.748893] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10760.166638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10760.178172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10760.193614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10762.209939] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10762.211698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10762.216828] Lustre: Skipped 1 previous similar message [10762.220651] Lustre: Skipped 1 previous similar message [10763.517090] Lustre: server umount lustre-MDT0000 complete [10767.633202] LustreError: 252688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787879110 with bad export cookie 12563708731082572921 [10767.644883] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10767.650733] LustreError: 252688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10768.176531] Lustre: server umount lustre-MDT0001 complete [10782.302868] Lustre: server umount lustre-OST0000 complete [10795.370239] Lustre: server umount lustre-OST0001 complete [10810.189639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10820.879173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10829.884626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10840.064704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10864.432368] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [10879.366210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10879.377743] Lustre: lustre-MDT0000: reset Object Index mappings [10879.646456] LustreError: 260216:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10879.658115] LustreError: 260216:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [10879.756814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10884.418755] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [10885.090400] LustreError: 260217:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10889.188840] LustreError: 260216:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10894.125569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10894.149292] Lustre: lustre-MDT0001: reset Object Index mappings [10894.305645] LustreError: 260217:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10894.738462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10899.376708] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [10902.305334] Lustre: 261358:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10907.905825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10914.379253] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [10919.466117] LustreError: 261713:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10923.229642] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10923.483954] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10923.486705] Lustre: Skipped 1 previous similar message [10924.543803] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10924.546699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10932.642539] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [10945.092256] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10950.220117] Lustre: 263234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10956.273981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10956.278517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10956.290693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10956.296171] Lustre: Skipped 2 previous similar messages [10957.966116] Lustre: server umount lustre-MDT0000 complete [10960.366570] LustreError: 260216:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10960.371541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10960.389274] Lustre: Skipped 1 previous similar message [10962.673924] LustreError: 260196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787879305 with bad export cookie 12563708731082579711 [10962.687516] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10962.694658] LustreError: 260196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10963.216440] Lustre: server umount lustre-MDT0001 complete [10977.434051] Lustre: server umount lustre-OST0000 complete [10981.354345] Lustre: 250199:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787879308/real 1787879308] req@ffff8ca4cd5eb480 x1874726947523200/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787879324 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10981.424420] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10981.438620] Lustre: Skipped 1 previous similar message [10981.785775] Lustre: server umount lustre-OST0001 complete [11002.916221] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 21:09:04 (1787879344) [11017.951642] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [11033.933292] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [11084.755085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11085.120117] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11085.204525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11085.367849] Lustre: lustre-MDT0000: new disk, initializing [11085.596873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11085.642781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11092.318632] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11104.469425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11104.651369] Lustre: 267510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11104.778595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11104.787598] Lustre: Skipped 1 previous similar message [11104.957725] Lustre: lustre-MDT0001: new disk, initializing [11105.170563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11105.183292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11110.491738] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11122.694852] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11130.617495] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11139.112617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11139.337433] Lustre: lustre-OST0000: new disk, initializing [11139.343646] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11139.358269] Lustre: 269477:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11141.270421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11141.289028] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11141.488851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11146.168335] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11155.903190] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11161.517763] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11161.716987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11166.880615] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11167.095553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11172.322692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11172.328571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11172.336682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11172.343569] Lustre: Skipped 1 previous similar message [11172.356570] Lustre: Skipped 1 previous similar message [11175.211630] Lustre: server umount lustre-OST0000 complete [11182.053688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11182.073898] LustreError: Skipped 1 previous similar message [11182.083722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11185.588889] Lustre: server umount lustre-MDT0000 complete [11187.689297] LustreError: 267516:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11187.723090] LustreError: 267516:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [11191.551072] LustreError: 267502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787879534 with bad export cookie 12563708731082581804 [11191.554654] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11191.565689] LustreError: 267502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11191.966782] Lustre: server umount lustre-MDT0001 complete [11214.584683] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [11225.879681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11226.276187] LustreError: 272429:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11226.320632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11226.325670] Lustre: Skipped 2 previous similar messages [11230.923975] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11238.574451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11244.461762] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11248.191101] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11258.269545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11267.836428] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11276.806172] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11277.014100] Lustre: lustre-OST0001: new disk, initializing [11277.025689] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11277.034311] Lustre: 274739:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11278.544648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11278.566672] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11278.705600] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11283.675693] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11292.608460] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11295.990283] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11301.362559] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11301.411186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11309.538407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11309.554210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11309.566347] Lustre: Skipped 5 previous similar messages [11309.577707] Lustre: Skipped 4 previous similar messages [11315.026178] Lustre: server umount lustre-MDT0000 complete [11318.774082] LustreError: 272409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787879661 with bad export cookie 12563708731082582686 [11318.782688] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11318.803324] LustreError: 272409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11319.145215] Lustre: server umount lustre-MDT0001 complete [11332.473157] Lustre: server umount lustre-OST0000 complete [11335.650314] Lustre: 250201:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787879662/real 1787879662] req@ffff8ca4c2394b40 x1874726947633536/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787879678 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11336.161748] Lustre: server umount lustre-OST0001 complete [11359.137794] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [11372.035035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11372.725510] LustreError: 277796:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11372.752151] LustreError: 277796:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11372.810159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11372.820110] Lustre: Skipped 3 previous similar messages [11379.091112] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11390.188487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11395.995478] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11399.666871] Lustre: 278938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11407.978327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11415.218515] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11418.707417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11424.857359] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11430.420956] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11434.337293] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11444.771088] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11449.087828] Lustre: 280815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11455.031894] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11455.056558] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11464.981359] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 21:16:46 (1787879806) [11481.583551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11481.602196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11481.609027] Lustre: Skipped 3 previous similar messages [11481.622063] Lustre: Skipped 7 previous similar messages [11485.721601] Lustre: server umount lustre-MDT0000 complete [11488.852440] LustreError: 277778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787879831 with bad export cookie 12563708731082589238 [11488.864269] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11488.866582] LustreError: 277778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11495.184267] Lustre: server umount lustre-MDT0001 complete [11504.630165] Lustre: server umount lustre-OST0000 complete [11508.635664] Lustre: server umount lustre-OST0001 complete [11515.552500] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [11525.107288] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [11568.121854] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [11578.665987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11578.973434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11578.991679] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11579.119479] Lustre: lustre-MDT0000: new disk, initializing [11579.203237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11584.882956] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11596.639068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11596.722512] Lustre: 285058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11596.973336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11601.683830] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11614.084853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11614.344701] Lustre: 286690:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11616.399245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11620.395336] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11631.305063] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11631.436409] Lustre: 287563:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11631.537538] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11631.552417] Lustre: Skipped 6 previous similar messages [11633.024415] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11633.030311] Lustre: Skipped 1 previous similar message [11633.144939] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11638.128642] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11648.048117] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11651.794039] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11651.799126] Lustre: Skipped 1 previous similar message [11666.016730] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 21:20:07 (1787880007) [11668.970162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11668.971942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11668.972907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11668.982542] Lustre: Skipped 5 previous similar messages [11669.001921] Lustre: Skipped 7 previous similar messages [11674.014642] Lustre: server umount lustre-MDT0000 complete [11674.084604] LustreError: 285063:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11674.111482] LustreError: 285063:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 19 previous similar messages [11677.500530] LustreError: 285050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787880020 with bad export cookie 12563708731082596882 [11677.506105] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11677.520896] LustreError: 285050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11677.861204] Lustre: server umount lustre-MDT0001 complete [11691.774255] Lustre: server umount lustre-OST0000 complete [11705.852479] Lustre: server umount lustre-OST0001 complete [11714.208497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11721.061203] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11732.712267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11738.950486] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11748.974877] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11755.498590] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11762.752646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11769.679446] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11775.599661] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11803.115670] loop1: detected capacity change from 0 to 204800000 [11803.408665] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11803.421617] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11803.431784] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11809.282427] loop1: detected capacity change from 0 to 204800000 [11815.755853] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11815.890769] LustreError: 289796:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs9007: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11815.927850] LustreError: 289796:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs9007 [11815.939994] LustreError: 289796:0:(mgs_handler.c:517:mgs_target_reg()) fs9007-MDT0001: Error checking target index: rc = -13 [11815.956697] LustreError: fs9007-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11815.963904] LustreError: 293550:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11815.970313] LustreError: 293550:0:(tgt_mount.c:2129:server_put_super()) no obd fs9007-MDT0001 [11815.975804] LustreError: 293550:0:(tgt_mount.c:116:server_deregister_mount()) fs9007-MDT0001 not registered [11816.009743] Lustre: server umount fs9007-MDT0001 complete [11816.011686] LustreError: 293550:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11823.320535] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11823.401513] Lustre: Setting parameter fs9007-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs9007-MDT0001 [11823.440846] Lustre: srv-fs9007-MDT0001: No data found on store. Initialize space. [11823.444081] Lustre: Skipped 4 previous similar messages [11823.549381] Lustre: fs9007-MDT0001: new disk, initializing [11823.552665] Lustre: Skipped 3 previous similar messages [11823.665470] Lustre: srv-fs9007-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11823.677185] Lustre: 294017:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9007-MDT0001-mdtlov: can't create new llog: rc = -115 [11823.690639] LustreError: 294017:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9007-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11827.831476] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11834.594255] Lustre: server umount fs9007-MDT0001 complete [11850.893606] loop1: detected capacity change from 0 to 204800000 [11856.386145] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11857.871823] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11858.155616] Lustre: 295586:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9007-MDT0001-mdtlov: can't create new llog: rc = -115 [11858.163910] LustreError: 295586:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9007-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11862.233977] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11866.797761] Lustre: server umount fs9007-MDT0001 complete [11876.839733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11876.843126] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11876.843317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11876.848692] Lustre: Skipped 2 previous similar messages [11876.867234] LustreError: Skipped 1 previous similar message [11882.031495] Lustre: server umount lustre-MDT0000 complete [11890.663358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11890.780716] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11895.290962] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11896.324904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11902.214152] loop1: detected capacity change from 0 to 204800000 [11906.960749] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11907.542915] Lustre: 298031:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs9007-MDT0001-mdtlov: can't create new llog: rc = -115 [11907.553192] LustreError: 298031:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs9007-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11912.469688] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [11918.391758] Lustre: server umount fs9007-MDT0001 complete [11932.139969] LustreError: 291638:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11932.154741] LustreError: 291638:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 101 previous similar messages [11942.382867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11942.393522] LustreError: Skipped 1 previous similar message [11946.929623] Lustre: server umount lustre-OST0000 complete [11956.663901] Lustre: server umount lustre-MDT0000 complete [11960.346944] LustreError: 289789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787880303 with bad export cookie 12563708731082599570 [11960.356946] LustreError: 289789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11960.365101] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11960.623570] Lustre: server umount lustre-MDT0001 complete [11971.454788] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [11974.847644] Key type lgssc unregistered [11975.183592] LNet: 300379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11975.201703] LNetError: 300379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11975.233630] LNet: Removed LNI 192.168.204.117@tcp [11976.198178] Key type .llcrypt unregistered [11976.203756] Key type ._llcrypt unregistered [11998.350504] Key type ._llcrypt registered [11998.354307] Key type .llcrypt registered [11998.482895] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [11999.751566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11999.911865] alg: No test for adler32 (adler32-zlib) [12000.958914] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [12001.214405] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [12002.873190] Key type lgssc registered [12003.888494] Lustre: Echo OBD driver; http://www.lustre.org/ [12010.753258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12010.766271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12012.291409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12018.037649] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [12022.765914] LustreError: 301652:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12022.791809] LustreError: 301652:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12027.875842] LustreError: 301651:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12030.355456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12031.381423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12036.730691] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [12044.564379] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12051.055505] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12056.631040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12056.969457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12058.026181] LustreError: 303574:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12058.062372] LustreError: 303574:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12063.211827] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [12063.215308] LustreError: 303574:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12063.236820] LustreError: 303574:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12068.326840] LustreError: 303484:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12068.347015] LustreError: 303484:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [12070.565921] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12078.465444] LustreError: 303483:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12078.507517] LustreError: 303483:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12088.600693] loop1: detected capacity change from 0 to 204800000 [12088.832609] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12088.840473] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12088.860927] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12093.742913] loop1: detected capacity change from 0 to 204800000 [12098.920550] LustreError: 303575:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12098.931795] LustreError: 303575:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [12099.340526] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12099.508249] Lustre: fs22714-OST0000: new disk, initializing [12099.511471] Lustre: srv-fs22714-OST0000: No data found on store. Initialize space. [12099.516556] Lustre: 305388:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12099.585669] Lustre: fs22714-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12106.942121] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [12116.406854] Lustre: server umount fs22714-OST0000 complete [12129.315306] loop1: detected capacity change from 0 to 204800000 [12134.755930] LustreError: 303483:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12134.775062] LustreError: 303483:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 20 previous similar messages [12136.738951] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12137.154581] Lustre: fs22714-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12140.029453] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12144.597552] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [12145.130461] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12153.713524] loop2: detected capacity change from 0 to 204800000 [12153.861104] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12153.881944] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12153.895519] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12158.671720] loop2: detected capacity change from 0 to 204800000 [12163.806282] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12163.847843] LustreError: fs22714-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12163.856299] LustreError: fs22714-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12163.863780] LustreError: 308431:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [12163.869474] LustreError: 308431:0:(tgt_mount.c:2129:server_put_super()) no obd fs22714-OST0001 [12163.873558] LustreError: 308431:0:(tgt_mount.c:116:server_deregister_mount()) fs22714-OST0001 not registered [12163.900763] Lustre: server umount fs22714-OST0001 complete [12163.902648] LustreError: 308431:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12177.246911] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12177.354409] Lustre: fs22714-OST0001: new disk, initializing [12177.360721] Lustre: srv-fs22714-OST0001: No data found on store. Initialize space. [12177.366420] Lustre: 308984:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12177.462381] Lustre: fs22714-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12184.565950] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [12192.973461] Lustre: server umount fs22714-OST0000 complete [12201.317714] LustreError: 303481:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12201.345876] LustreError: 303481:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 38 previous similar messages [12202.759100] Lustre: server umount fs22714-OST0001 complete [12218.429553] loop1: detected capacity change from 0 to 204800000 [12224.691449] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12225.270744] Lustre: fs22714-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12232.976762] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [12239.926360] loop2: detected capacity change from 0 to 204800000 [12244.846753] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12252.598289] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [12264.515746] Lustre: server umount fs22714-OST0000 complete [12275.641357] Lustre: server umount fs22714-OST0001 complete [12291.556439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12291.559555] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12291.595529] Lustre: Skipped 1 previous similar message [12291.602448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12291.618914] Lustre: Skipped 1 previous similar message [12296.676351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12296.679944] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12296.955849] Lustre: server umount lustre-OST0000 complete [12301.794624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12301.808345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12301.819827] Lustre: Skipped 2 previous similar messages [12306.885387] Lustre: server umount lustre-MDT0000 complete [12311.617956] LustreError: 301631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787880654 with bad export cookie 13146265264439495817 [12311.626203] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12311.631254] LustreError: 301631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12311.986559] Lustre: server umount lustre-MDT0001 complete [12324.127821] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [12326.297719] Key type lgssc unregistered [12326.523266] LNet: 314910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12326.529240] LNetError: 314910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12326.552187] LNet: Removed LNI 192.168.204.117@tcp [12327.553377] Key type .llcrypt unregistered [12327.555608] Key type ._llcrypt unregistered [12361.882858] Key type ._llcrypt registered [12361.884847] Key type .llcrypt registered [12362.020414] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [12376.849491] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [12377.341934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12377.362022] alg: No test for adler32 (adler32-zlib) [12378.298956] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [12378.502253] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [12380.168318] Key type lgssc registered [12381.315611] Lustre: Echo OBD driver; http://www.lustre.org/ [12420.272806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12420.326053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12421.649369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12421.682669] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12421.829708] Lustre: lustre-MDT0000: new disk, initializing [12421.917145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12421.945404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12428.238358] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [12441.990116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12442.185903] Lustre: 318772:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12442.274637] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12442.286913] Lustre: Skipped 1 previous similar message [12442.408060] Lustre: lustre-MDT0001: new disk, initializing [12442.535133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12442.589307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12442.609892] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12447.783593] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [12457.640670] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12464.832799] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12473.617337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12474.080235] Lustre: lustre-OST0000: new disk, initializing [12474.088087] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12474.123130] Lustre: 320744:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12474.322094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12479.564028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12479.570021] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12479.692211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12481.427870] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [12491.649190] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12494.327361] Lustre: Mounted lustre-client - version 2.17.57_99_gb2219ab [12495.390525] Lustre: Unmounted lustre-client [12509.668218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12509.675665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12509.695360] Lustre: Skipped 1 previous similar message [12509.716048] Lustre: Skipped 1 previous similar message [12511.978821] Lustre: server umount lustre-OST0000 complete [12519.395163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12519.402772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12519.430779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12522.623701] Lustre: server umount lustre-MDT0000 complete [12526.655721] LustreError: 318763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787880869 with bad export cookie 16377658865403989334 [12526.667479] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12526.960300] Lustre: server umount lustre-MDT0001 complete [12537.026645] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [12539.930511] Key type lgssc unregistered [12548.451519] LNet: 322905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12548.469887] LNetError: 322905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12625.323922] LNet: Removed LNI 192.168.204.117@tcp [12626.474249] Key type .llcrypt unregistered [12626.476895] Key type ._llcrypt unregistered [12647.727592] Key type ._llcrypt registered [12647.731598] Key type .llcrypt registered [12647.841784] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [12664.955464] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [12666.046454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12666.085804] alg: No test for adler32 (adler32-zlib) [12667.130306] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [12667.420895] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [12669.088239] Key type lgssc registered [12670.421541] Lustre: Echo OBD driver; http://www.lustre.org/ [12706.897384] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 21:37:28 (1787881048) [12715.655841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12715.695750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12716.907382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12716.932929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12716.981872] Lustre: lustre-MDT0000: new disk, initializing [12717.036775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12717.062984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12722.719312] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [12736.328937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12736.558154] Lustre: 326129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12736.656052] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12736.669961] Lustre: Skipped 1 previous similar message [12736.784207] Lustre: lustre-MDT0001: new disk, initializing [12736.939697] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12736.986035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12737.013823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12742.445928] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [12754.558899] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12762.002316] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12771.139577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12771.447410] Lustre: lustre-OST0000: new disk, initializing [12771.453153] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12771.461894] Lustre: 328102:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12771.594950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12778.872963] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [12780.654166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12780.676609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12780.813322] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12789.084773] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12816.867844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12816.869520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12816.880752] LustreError: Skipped 1 previous similar message [12816.882598] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12816.924194] Lustre: Skipped 1 previous similar message [12818.483973] Lustre: server umount lustre-OST0000 complete [12824.033556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12824.039528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12824.052171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12824.056661] Lustre: Skipped 1 previous similar message [12827.109659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12827.117056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12829.423897] Lustre: server umount lustre-MDT0000 complete [12834.792155] LustreError: 326120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787881177 with bad export cookie 576038778611178800 [12834.802123] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12834.808457] LustreError: 326120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12835.047766] Lustre: server umount lustre-MDT0001 complete [12849.060170] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [12852.756789] Key type lgssc unregistered [12853.223671] LNet: 330027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12853.234914] LNetError: 330027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12853.259618] LNet: Removed LNI 192.168.204.117@tcp [12854.207853] Key type .llcrypt unregistered [12854.210446] Key type ._llcrypt unregistered [12879.263798] Key type ._llcrypt registered [12879.267526] Key type .llcrypt registered [12879.403194] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [12880.722158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12880.890167] alg: No test for adler32 (adler32-zlib) [12881.972438] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [12882.250217] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [12883.961493] Key type lgssc registered [12885.156507] Lustre: Echo OBD driver; http://www.lustre.org/ [12893.188452] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12893.221030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12894.886084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12899.354349] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [12904.435234] LustreError: 331302:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12904.460340] LustreError: 331302:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12908.163719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12908.587404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12912.191143] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [12920.204920] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12927.750380] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12935.087626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12935.519713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12942.761208] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [12954.452124] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12965.501687] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12967.143373] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12971.938248] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.17@tcp (stopping) [12974.563764] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12974.573128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12974.578630] Lustre: Skipped 2 previous similar messages [12974.584407] Lustre: Skipped 1 previous similar message [12977.006080] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.17@tcp (stopping) [12977.020368] Lustre: Skipped 1 previous similar message [12978.102459] Lustre: server umount lustre-MDT0000 complete [12979.683123] LustreError: 331298:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12979.692420] LustreError: 331298:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12981.792209] LustreError: 331284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787881324 with bad export cookie 14454415242674913258 [12981.804281] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12981.811827] LustreError: 331284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12982.114198] LustreError: 331297:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.17@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12982.155213] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.17@tcp (stopping) [12984.802538] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12984.805741] LustreError: 333704:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12986.849735] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12986.854973] Lustre: Skipped 1 previous similar message [12988.099976] Lustre: server umount lustre-MDT0001 complete [12992.548324] Lustre: server umount lustre-OST0000 complete [13003.093446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13008.437970] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [13011.513593] Lustre: 335486:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.117@tcp [13014.338821] Lustre: server umount MGS complete [13024.174674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13024.666081] LustreError: 336058:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13024.691361] LustreError: 336058:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [13024.754033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13029.608565] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [13034.977507] LustreError: 336059:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13034.987110] LustreError: 336059:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [13038.658048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13039.245565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13044.568685] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [13053.662388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13053.889923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13059.729734] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [13067.231553] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13069.132549] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13074.569179] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13075.927340] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13080.546464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13080.554548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13080.568453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13080.572837] Lustre: Skipped 1 previous similar message [13082.429343] Lustre: server umount lustre-OST0000 complete [13089.762716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13089.786760] Lustre: Skipped 1 previous similar message [13095.978143] Lustre: server umount lustre-MDT0000 complete [13099.862536] LustreError: 336038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787881442 with bad export cookie 14454415242674914098 [13099.866918] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13099.880176] LustreError: 336038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13100.245766] Lustre: server umount lustre-MDT0001 complete [13112.055938] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [13114.989351] Key type lgssc unregistered [13115.275647] LNet: 339500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13115.294384] LNetError: 339500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13115.322315] LNet: Removed LNI 192.168.204.117@tcp [13116.221755] Key type .llcrypt unregistered [13116.223831] Key type ._llcrypt unregistered [13135.231581] Key type ._llcrypt registered [13135.235794] Key type .llcrypt registered [13135.396086] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [13152.683348] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [13154.232598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13154.373048] alg: No test for adler32 (adler32-zlib) [13155.489920] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [13155.847195] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [13157.576609] Key type lgssc registered [13158.872750] Lustre: Echo OBD driver; http://www.lustre.org/ [13199.558248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13199.646526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13201.061314] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13201.096328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13201.252502] Lustre: lustre-MDT0000: new disk, initializing [13201.387814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13201.411544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13206.810558] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [13218.429200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13218.586114] Lustre: 342663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13218.658489] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13218.662597] Lustre: Skipped 1 previous similar message [13218.772410] Lustre: lustre-MDT0001: new disk, initializing [13218.944237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13219.002943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13219.016934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13224.798878] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [13234.835991] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13242.139957] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13251.070422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13251.415656] Lustre: lustre-OST0000: new disk, initializing [13251.421660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13251.435230] Lustre: 344632:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13251.593722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13257.652958] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [13259.340588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13259.347732] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13259.469069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13268.550361] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13274.198250] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13274.427129] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13279.755694] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13280.012806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13285.358645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13285.374344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13285.403488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13285.406105] Lustre: Skipped 1 previous similar message [13285.434421] Lustre: Skipped 1 previous similar message [13288.636181] Lustre: server umount lustre-OST0000 complete [13295.589550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13295.593676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13295.604779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13295.615186] LustreError: Skipped 1 previous similar message [13295.624280] Lustre: Skipped 1 previous similar message [13299.496520] Lustre: server umount lustre-MDT0000 complete [13303.699421] LustreError: 344631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787881646 with bad export cookie 3916279624868345691 [13303.708783] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13303.710896] LustreError: 344631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13304.067786] Lustre: server umount lustre-MDT0001 complete [13313.369653] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 21:47:34 (1787881654) [13333.175389] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [13344.241676] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [13388.613286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13388.911646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13388.949255] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13389.066463] Lustre: lustre-MDT0000: new disk, initializing [13389.229100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13389.257669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13394.823796] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [13406.667184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13406.784767] Lustre: 349395:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13406.848286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13406.853881] Lustre: Skipped 1 previous similar message [13406.966903] Lustre: lustre-MDT0001: new disk, initializing [13407.059223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13407.087924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13407.100791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13411.926394] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [13423.119747] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13430.301391] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13437.919456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13438.313583] Lustre: lustre-OST0000: new disk, initializing [13438.319489] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13438.330323] Lustre: 351364:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13438.436044] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13444.660450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13444.678378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13444.767164] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13444.855177] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [13457.232128] Lustre: DEBUG MARKER: oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13501.413188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13501.413882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13501.431826] LustreError: Skipped 1 previous similar message [13501.439550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13501.473755] Lustre: Skipped 2 previous similar messages [13506.531740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13506.537127] Lustre: Skipped 2 previous similar messages [13511.654996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13511.665358] Lustre: Skipped 1 previous similar message [13513.185280] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13513.455176] Lustre: server umount lustre-OST0000 complete [13519.849394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13519.860659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13519.873355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13523.938534] Lustre: server umount lustre-MDT0000 complete [13527.009658] LustreError: 349407:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13527.036649] LustreError: 349407:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [13528.655978] LustreError: 349389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787881871 with bad export cookie 3916279624868346566 [13528.661829] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13528.668060] LustreError: 349389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13529.142224] Lustre: server umount lustre-MDT0001 complete [13542.176366] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [13545.785526] Key type lgssc unregistered [13546.111836] LNet: 353545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13546.118202] LNetError: 353545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13546.136776] LNet: Removed LNI 192.168.204.117@tcp [13547.120508] Key type .llcrypt unregistered [13547.124250] Key type ._llcrypt unregistered [13649.115962] Key type ._llcrypt registered [13649.118284] Key type .llcrypt registered [13649.213392] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [13649.477498] Key type .llcrypt unregistered [13649.479933] Key type ._llcrypt unregistered [13732.777267] Key type ._llcrypt registered [13732.784161] Key type .llcrypt registered [13732.929723] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [13733.361559] Key type .llcrypt unregistered [13733.367125] Key type ._llcrypt unregistered [13766.330242] Key type ._llcrypt registered [13766.332143] Key type .llcrypt registered [13766.431357] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [13766.749709] Key type .llcrypt unregistered [13766.755564] Key type ._llcrypt unregistered [13796.780714] Key type ._llcrypt registered [13796.784828] Key type .llcrypt registered [13796.900098] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [13797.242325] Key type .llcrypt unregistered [13797.245781] Key type ._llcrypt unregistered [13876.558984] Key type ._llcrypt registered [13876.566142] Key type .llcrypt registered [13876.696466] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [13877.029460] Key type .llcrypt unregistered [13877.032483] Key type ._llcrypt unregistered [13907.227192] Key type ._llcrypt registered [13907.231450] Key type .llcrypt registered [13907.366270] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [13907.873441] Key type .llcrypt unregistered [13907.875748] Key type ._llcrypt unregistered [13936.342324] Key type ._llcrypt registered [13936.345800] Key type .llcrypt registered [13936.436430] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [13936.685709] Key type .llcrypt unregistered [13936.689401] Key type ._llcrypt unregistered [13968.682220] Key type ._llcrypt registered [13968.693754] Key type .llcrypt registered [13968.877307] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [13969.337381] Key type .llcrypt unregistered [13969.341600] Key type ._llcrypt unregistered [14000.636156] Key type ._llcrypt registered [14000.644161] Key type .llcrypt registered [14000.759268] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [14001.183454] Key type .llcrypt unregistered [14001.187954] Key type ._llcrypt unregistered [14032.430787] Key type ._llcrypt registered [14032.438872] Key type .llcrypt registered [14032.559926] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [14032.864510] Key type .llcrypt unregistered [14032.868128] Key type ._llcrypt unregistered [14098.974807] Key type ._llcrypt registered [14098.976354] Key type .llcrypt registered [14099.137685] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [14099.572516] Key type .llcrypt unregistered [14099.574941] Key type ._llcrypt unregistered [14132.006702] Key type ._llcrypt registered [14132.009131] Key type .llcrypt registered [14132.139567] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [14132.590325] Key type .llcrypt unregistered [14132.599219] Key type ._llcrypt unregistered [14167.155656] Key type ._llcrypt registered [14167.159445] Key type .llcrypt registered [14167.268544] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [14167.784610] Key type .llcrypt unregistered [14167.786531] Key type ._llcrypt unregistered [14217.827481] Key type ._llcrypt registered [14217.830104] Key type .llcrypt registered [14217.961485] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [14232.385752] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [14233.844331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14233.891152] alg: No test for adler32 (adler32-zlib) [14235.153926] Lustre: Lustre: Build Version: 2.17.57_99_gb2219ab [14235.357416] LNet: Added LNI 192.168.204.117@tcp [8/256/0/180] [14237.096377] Key type lgssc registered [14238.455514] Lustre: Echo OBD driver; http://www.lustre.org/ [14273.847434] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14273.871763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14275.109198] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14275.153205] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14275.215801] Lustre: tst250fs-MDT0000: new disk, initializing [14275.300670] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14275.315954] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14279.533940] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [14289.939711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14290.089178] Lustre: 373875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14290.143151] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14290.146864] Lustre: Skipped 1 previous similar message [14290.260051] Lustre: tst250fs-MDT0001: new disk, initializing [14290.383524] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14290.427535] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14290.442990] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14295.516265] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [14300.970846] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14308.523988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14308.830312] Lustre: tst250fs-OST0000: new disk, initializing [14308.836506] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14308.843082] Lustre: 375504:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14308.951825] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14315.581406] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14315.595416] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14315.694667] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [14315.713240] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14327.062347] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14327.238465] Lustre: tst250fs-OST0001: new disk, initializing [14327.242916] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14327.248845] Lustre: 376378:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14327.329817] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14333.010524] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14333.021248] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14333.158132] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_default_debug -1 all [14333.158598] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14339.034139] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14341.172033] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14341.181136] Lustre: Skipped 1 previous similar message [14362.083332] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14362.087590] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14362.102545] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14364.130568] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14364.138405] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14367.674768] Lustre: server umount tst250fs-MDT0000 complete [14371.438646] LustreError: 373867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787882714 with bad export cookie 8562315961814183564 [14371.440060] LustreError: MGC192.168.204.117@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14371.478078] LustreError: 373867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14371.826560] Lustre: server umount tst250fs-MDT0001 complete [14385.959718] Lustre: server umount tst250fs-OST0000 complete [14399.347483] Lustre: server umount tst250fs-OST0001 complete [14419.950337] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [14428.856863] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [14466.292562] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 22:06:47 (1787882807) [14468.529872] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14487.444421] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing set_hostid [14497.519610] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing load_modules_local [14536.430315] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14221 sec ========= 22:07:56 (1787882876) [14538.642180] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:07:59 (1787882879) === [14543.535970] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:08:04 (1787882884) === [14593.417982] Lustre: DEBUG MARKER: oleg417-server.virtnet: executing unload_modules_local [14596.606493] Key type lgssc unregistered [14596.978761] LNet: 385529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14596.990685] LNetError: 385529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14597.021835] LNet: Removed LNI 192.168.204.117@tcp [14598.078235] Key type .llcrypt unregistered [14598.089320] Key type ._llcrypt unregistered