[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 590933245 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003281] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008411] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011128] LSM: Security Framework initializing [ 0.013043] Yama: becoming mindful. [ 0.014034] SELinux: Initializing. [ 0.015065] *** VALIDATE selinux *** [ 0.024318] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028839] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029165] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030106] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031115] *** VALIDATE tmpfs *** [ 0.032508] *** VALIDATE proc *** [ 0.033230] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.035248] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037133] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039032] Spectre V2 : User space: Vulnerable [ 0.041007] Speculative Store Bypass: Vulnerable [ 0.044477] debug: unmapping init [mem 0xffffffff9f459000-0xffffffff9f460fff] [ 0.047217] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048842] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049023] ... version: 2 [ 0.050014] ... bit width: 48 [ 0.051016] ... generic registers: 4 [ 0.052013] ... value mask: 0000ffffffffffff [ 0.053016] ... max period: 00007fffffffffff [ 0.054016] ... fixed-purpose events: 3 [ 0.055012] ... event mask: 000000070000000f [ 0.056304] rcu: Hierarchical SRCU implementation. [ 0.058782] smp: Bringing up secondary CPUs ... [ 0.059609] x86: Booting SMP configuration: [ 0.060030] .... node #0, CPUs: #1 #2 #3 [ 0.067017] smp: Brought up 1 node, 4 CPUs [ 0.069013] smpboot: Max logical packages: 1 [ 0.070014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.197056] node 0 deferred pages initialised in 123ms [ 0.200265] devtmpfs: initialized [ 0.202538] x86/mm: Memory block size: 128MB [ 0.206985] gcov: version magic: 0x41383552 [ 0.210388] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.211090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.213526] pinctrl core: initialized pinctrl subsystem [ 0.216358] [ 0.216952] ************************************************************* [ 0.220018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.223017] ** ** [ 0.225016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.228017] ** ** [ 0.230016] ** This means that this kernel is built to expose internal ** [ 0.233018] ** IOMMU data structures, which may compromise security on ** [ 0.235019] ** your system. ** [ 0.238020] ** ** [ 0.240018] ** If you see this message and you are not debugging the ** [ 0.243031] ** kernel, report this immediately to your vendor! ** [ 0.246018] ** ** [ 0.248015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.251019] ************************************************************* [ 0.253875] NET: Registered protocol family 16 [ 0.256505] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.259084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.261172] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.265144] cpuidle: using governor menu [ 0.267943] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.271846] PCI: Using configuration type 1 for base access [ 0.274229] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.286111] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.289107] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.293080] cryptd: max_cpu_qlen set to 1000 [ 0.296243] ACPI: Added _OSI(Module Device) [ 0.298017] ACPI: Added _OSI(Processor Device) [ 0.300019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.301014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.306032] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.313618] ACPI: Interpreter enabled [ 0.315077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.316105] ACPI: Using IOAPIC for interrupt routing [ 0.318222] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.322433] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.333548] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.336052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.339024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.342099] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.347303] acpiphp: Slot [2] registered [ 0.349136] acpiphp: Slot [5] registered [ 0.350123] acpiphp: Slot [6] registered [ 0.352129] acpiphp: Slot [7] registered [ 0.354184] acpiphp: Slot [8] registered [ 0.355137] acpiphp: Slot [9] registered [ 0.357158] acpiphp: Slot [10] registered [ 0.358295] acpiphp: Slot [3] registered [ 0.360172] acpiphp: Slot [4] registered [ 0.362135] acpiphp: Slot [11] registered [ 0.363165] acpiphp: Slot [12] registered [ 0.365253] acpiphp: Slot [13] registered [ 0.367147] acpiphp: Slot [14] registered [ 0.368119] acpiphp: Slot [15] registered [ 0.370114] acpiphp: Slot [16] registered [ 0.371132] acpiphp: Slot [17] registered [ 0.373130] acpiphp: Slot [18] registered [ 0.374101] acpiphp: Slot [19] registered [ 0.376099] acpiphp: Slot [20] registered [ 0.378122] acpiphp: Slot [21] registered [ 0.380151] acpiphp: Slot [22] registered [ 0.382121] acpiphp: Slot [23] registered [ 0.384152] acpiphp: Slot [24] registered [ 0.386140] acpiphp: Slot [25] registered [ 0.388129] acpiphp: Slot [26] registered [ 0.391186] acpiphp: Slot [27] registered [ 0.392152] acpiphp: Slot [28] registered [ 0.394149] acpiphp: Slot [29] registered [ 0.395213] acpiphp: Slot [30] registered [ 0.397172] acpiphp: Slot [31] registered [ 0.398067] PCI host bridge to bus 0000:00 [ 0.399025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.401031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.403032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.406051] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.409047] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.411036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.413191] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.416352] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.420393] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.432873] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.437063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.440039] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.442021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.443089] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.446414] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.449209] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.451088] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.455059] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.460025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.473024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.480022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.485523] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.498017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.510016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.534020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.547569] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.555066] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.562025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.583022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.593698] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.602015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.610015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.631015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.644074] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.658019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.667017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.697015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.715320] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.727022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.737020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.764016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.777907] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.785018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.797016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.827019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.841657] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.845579] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.849471] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.853531] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.857257] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.862040] iommu: Default domain type: Passthrough [ 0.864643] SCSI subsystem initialized [ 0.866228] ACPI: bus type USB registered [ 0.868147] usbcore: registered new interface driver usbfs [ 0.870144] usbcore: registered new interface driver hub [ 0.873119] usbcore: registered new device driver usb [ 0.875239] pps_core: LinuxPPS API ver. 1 registered [ 0.877019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.882094] PTP clock support registered [ 0.884239] EDAC MC: Ver: 3.0.0 [ 0.887180] PCI: Using ACPI for IRQ routing [ 0.888913] NetLabel: Initializing [ 0.891016] NetLabel: domain hash size = 128 [ 0.892012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.894081] NetLabel: unlabeled traffic allowed by default [ 0.897158] vgaarb: loaded [ 0.899333] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.901016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.909005] clocksource: Switched to clocksource kvm-clock [ 1.027644] VFS: Disk quotas dquot_6.6.0 [ 1.030109] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.032967] *** VALIDATE ramfs *** [ 1.034558] *** VALIDATE hugetlbfs *** [ 1.036488] pnp: PnP ACPI init [ 1.039430] pnp: PnP ACPI: found 6 devices [ 1.060683] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.065866] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.068658] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.071280] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.074194] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.076978] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.080698] NET: Registered protocol family 2 [ 1.083677] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.089675] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.093567] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.099386] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.103183] TCP: Hash tables configured (established 65536 bind 65536) [ 1.106834] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.110865] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.114132] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.118572] NET: Registered protocol family 1 [ 1.121744] RPC: Registered named UNIX socket transport module. [ 1.124040] RPC: Registered udp transport module. [ 1.126485] RPC: Registered tcp transport module. [ 1.128695] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.131706] NET: Registered protocol family 44 [ 1.133578] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.136742] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.139974] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.143217] PCI: CLS 0 bytes, default 64 [ 1.145790] Unpacking initramfs... [ 2.606548] debug: unmapping init [mem 0xffff8bc73cc54000-0xffff8bc73ffbffff] [ 2.610680] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.613244] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.616192] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.132954] Initialise system trusted keyrings [ 3.134761] Key type blacklist registered [ 3.136924] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.145620] zbud: loaded [ 3.148831] *** VALIDATE nfs *** [ 3.150279] *** VALIDATE nfs4 *** [ 3.152317] pstore: using deflate compression [ 3.157420] Platform Keyring initialized [ 3.287356] NET: Registered protocol family 38 [ 3.289505] Key type asymmetric registered [ 3.291383] Asymmetric key parser 'x509' registered [ 3.293483] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.297301] io scheduler mq-deadline registered [ 3.299438] io scheduler kyber registered [ 3.301224] io scheduler bfq registered [ 3.303496] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.306943] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.309966] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.313458] ACPI: Power Button [PWRF] [ 3.319121] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.326635] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.353686] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.365302] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.415113] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.447091] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.476441] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.483761] Non-volatile memory driver v1.3 [ 3.485979] Linux agpgart interface v0.103 [ 3.523747] virtio_blk virtio1: [vda] 146872 512-byte logical blocks (75.2 MB/71.7 MiB) [ 3.527080] vda: detected capacity change from 0 to 75198464 [ 3.545917] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.549501] vdb: detected capacity change from 0 to 1073741824 [ 3.566619] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.569465] vdc: detected capacity change from 0 to 2621440000 [ 3.584365] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.587681] vdd: detected capacity change from 0 to 2621440000 [ 3.604705] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.609204] vde: detected capacity change from 0 to 4294967296 [ 3.630551] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.633985] vdf: detected capacity change from 0 to 4294967296 [ 3.642804] libphy: Fixed MDIO Bus: probed [ 3.651452] usbcore: registered new interface driver usbserial_generic [ 3.654669] usbserial: USB Serial support registered for generic [ 3.659045] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.664552] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.666806] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.670353] mousedev: PS/2 mouse device common for all mice [ 3.673624] rtc_cmos 00:05: RTC can wake from S4 [ 3.677693] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.682620] rtc_cmos 00:05: registered as rtc0 [ 3.685713] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.685833] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.688893] intel_pstate: CPU model not supported [ 3.695662] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.702405] hid: raw HID events driver (C) Jiri Kosina [ 3.705148] usbcore: registered new interface driver usbhid [ 3.707454] usbhid: USB HID core driver [ 3.709217] drop_monitor: Initializing network drop monitor service [ 3.711607] Initializing XFRM netlink socket [ 3.713806] NET: Registered protocol family 10 [ 3.716671] Segment Routing with IPv6 [ 3.718462] NET: Registered protocol family 17 [ 3.721344] mpls_gso: MPLS GSO support [ 3.729107] RAS: Correctable Errors collector initialized. [ 3.731936] AVX version of gcm_enc/dec engaged. [ 3.733681] AES CTR mode by8 optimization enabled [ 3.826798] sched_clock: Marking stable (3826736405, 0)->(4807389912, -980653507) [ 3.830811] registered taskstats version 1 [ 3.833989] Loading compiled-in X.509 certificates [ 3.836965] zswap: loaded using pool lzo/zbud [ 3.869358] Key type big_key registered [ 3.882745] Key type encrypted registered [ 3.884608] ima: No TPM chip found, activating TPM-bypass! [ 3.887591] ima: Allocated hash algorithm: sha1 [ 3.889752] ima: No architecture policies found [ 3.891974] evm: Initialising EVM extended attributes: [ 3.895143] evm: security.selinux [ 3.896432] evm: security.ima [ 3.897663] evm: security.capability [ 3.898919] evm: HMAC attrs: 0x1 [ 3.901487] rtc_cmos 00:05: setting system clock to 2026-08-27 16:12:29 UTC (1787847149) [ 3.908190] debug: unmapping init [mem 0xffffffffa0403000-0xffffffffa05fffff] [ 3.911728] debug: unmapping init [mem 0xffffffff9f182000-0xffffffff9f458fff] [ 3.921301] Write protecting the kernel read-only data: 28672k [ 3.925434] debug: unmapping init [mem 0xffffffff9d803000-0xffffffff9d9fffff] [ 3.928960] debug: unmapping init [mem 0xffffffff9e114000-0xffffffff9e1fffff] [ 3.970371] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.980766] systemd[1]: Detected virtualization kvm. [ 3.983747] systemd[1]: Detected architecture x86-64. [ 3.985458] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.011229] systemd[1]: No hostname configured. [ 4.013101] systemd[1]: Set hostname to . [ 4.015681] random: systemd: uninitialized urandom read (16 bytes read) [ 4.018306] systemd[1]: Initializing machine ID from random generator. [ 4.082859] random: ln: uninitialized urandom read (6 bytes read) [ 4.185067] random: systemd: uninitialized urandom read (16 bytes read) [ 4.187732] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.192145] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.197353] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.921172] device-mapper: uevent: version 1.0.3 [ 4.924741] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.753385] virtio_net virtio0 ens2: renamed from eth0 [ 5.768863] random: fast init done [ 5.828272] scsi host0: ata_piix [ 5.881767] scsi host1: ata_piix [ 5.918088] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.920575] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.390528] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.403564] random: crng init done [ 10.404766] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.976161] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.187697] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.450537] SELinux: Disabled at runtime. [ 12.511654] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.521223] systemd[1]: Detected virtualization kvm. [ 12.523308] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.033439] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.041305] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.046185] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.051279] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.053835] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.060759] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.064165] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target RPC Port Mapper. [ OK [[ 13.122473] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.549076] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.873310] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.944474] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.076344] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.085938] EDAC sbridge: Ver: 1.1.2 [ 15.596270] Key type dns_resolver registered [ 15.911581] NFS: Registering the id_resolver key type [ 15.914872] Key type id_resolver registered [ 15.917719] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg409-server login: [ 56.567543] libcfs: loading out-of-tree module taints kernel. [ 56.640207] Key type ._llcrypt registered [ 56.646251] Key type .llcrypt registered [ 56.792207] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 75.211781] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 76.863761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 76.878976] alg: No test for adler32 (adler32-zlib) [ 78.289178] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 79.023829] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 80.735583] Key type lgssc registered [ 82.672219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 98.228330] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 144.858198] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 149.691037] hrtimer: interrupt took 4512585 ns [ 157.948925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 157.980395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.226653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 159.281081] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 159.425845] Lustre: lustre-MDT0000: new disk, initializing [ 159.488790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 159.512851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 164.699405] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.252380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 180.365312] Lustre: 6505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 180.408495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 180.413458] Lustre: Skipped 1 previous similar message [ 180.563040] Lustre: lustre-MDT0001: new disk, initializing [ 180.687956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 180.732958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 180.748894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 187.003502] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.197365] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 202.538182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.791552] Lustre: lustre-OST0000: new disk, initializing [ 202.794789] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 202.803920] Lustre: 8440:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.874345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 209.980464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 209.995146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 210.142752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 210.242820] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.804843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.953796] Lustre: lustre-OST0001: new disk, initializing [ 226.957531] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 226.963336] Lustre: 9513:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 227.056553] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 233.229514] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.137349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 237.152365] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 237.243361] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 246.765577] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 251.589568] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 257.550966] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing check_logdir /tmp/testlogs/ [ 264.526974] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing yml_node [ 268.621953] Lustre: DEBUG MARKER: Client: 2.17.57.86 [ 271.790290] Lustre: DEBUG MARKER: MDS: 2.17.57.86 [ 275.290720] Lustre: DEBUG MARKER: OSS: 2.17.57.86 [ 277.481375] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 12:17:01 EDT 2026 [ 297.365688] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 299.300835] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 318.944493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 318.954313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 318.980471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 319.472474] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.499350] Lustre: Skipped 1 previous similar message [ 319.504473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 319.515728] Lustre: Skipped 3 previous similar messages [ 322.630042] Lustre: server umount lustre-MDT0000 complete [ 326.735449] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847472 with bad export cookie 12659184985676625917 [ 326.739781] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 326.755559] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 329.697658] LustreError: 6510:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 329.698152] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.714534] LustreError: 6510:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 329.731205] Lustre: Skipped 1 previous similar message [ 329.761664] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 329.770574] Lustre: Skipped 1 previous similar message [ 331.744677] LustreError: 6510:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.776052] LustreError: 6510:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 333.356860] Lustre: server umount lustre-MDT0001 complete [ 338.439360] Lustre: server umount lustre-OST0000 complete [ 343.206848] Lustre: server umount lustre-OST0001 complete [ 351.936169] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 362.019872] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 403.016032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 403.358704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 403.428984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 403.513958] Lustre: lustre-MDT0000: new disk, initializing [ 403.585763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 403.603425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 408.771288] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 420.631582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.746577] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 420.801520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 420.807721] Lustre: Skipped 1 previous similar message [ 420.880968] Lustre: lustre-MDT0001: new disk, initializing [ 421.019529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 421.066525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 421.077874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 425.695636] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 436.018779] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 442.739680] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 450.146417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 450.488349] Lustre: lustre-OST0000: new disk, initializing [ 450.494068] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 450.499261] Lustre: 17040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 450.613745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 451.818154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 451.825955] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 451.919446] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 456.904779] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 465.269919] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 469.672403] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 469.982253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 474.891893] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 475.197844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 477.665957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.672635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 477.673185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 477.673190] Lustre: Skipped 2 previous similar messages [ 477.684262] Lustre: Skipped 2 previous similar messages [ 482.787882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 482.794937] Lustre: Skipped 2 previous similar messages [ 483.431587] Lustre: server umount lustre-OST0000 complete [ 487.906785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 487.912288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 487.928841] Lustre: Skipped 1 previous similar message [ 493.034882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 493.049607] Lustre: Skipped 3 previous similar messages [ 493.661437] Lustre: server umount lustre-MDT0000 complete [ 497.184462] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847642 with bad export cookie 12659184985676627450 [ 497.187377] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 497.195829] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 497.581407] Lustre: server umount lustre-MDT0001 complete [ 503.109965] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 12:20:47 (1787847647) [ 520.888490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.400955] LustreError: 19916:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.497820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 525.722826] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 526.820924] LustreError: 19917:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 531.939730] LustreError: 19917:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.958121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.247656] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 548.075465] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 553.691418] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 555.976615] Lustre: Failing over lustre-MDT0000 [ 556.002766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 556.009226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.017841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 556.129907] Lustre: server umount lustre-MDT0000 complete [ 557.024647] LustreError: 19917:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 557.053458] LustreError: 19917:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 573.407113] Lustre: 3638:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787847702/real 1787847702] req@ffff8bc7c04cb0c0 x1874693690913664/t0(0) o400->MGC192.168.204.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787847718 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 573.426556] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.440747] LustreError: 21156:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 573.458279] LustreError: 21156:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 577.834727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 583.648217] LustreError: 3635:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bc6876c7840 x1874693690916480/t0(0) o250->MGC192.168.204.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 589.271708] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 589.290939] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 589.294993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 589.327657] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 599.335118] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 603.622922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.625257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 603.634609] Lustre: Skipped 2 previous similar messages [ 603.652544] Lustre: Skipped 1 previous similar message [ 609.494391] Lustre: server umount lustre-MDT0000 complete [ 609.762600] LustreError: 19913:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.778766] LustreError: 19913:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 614.073679] LustreError: 19898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847759 with bad export cookie 12659184985676628864 [ 614.111105] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 614.508252] Lustre: server umount lustre-MDT0001 complete [ 624.382426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 625.263173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 625.276621] Lustre: Skipped 2 previous similar messages [ 630.997682] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 640.045263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 645.389694] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 653.999994] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 660.639717] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 663.975945] Lustre: Failing over lustre-MDT0000 [ 664.007691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 664.022132] Lustre: Skipped 1 previous similar message [ 664.446593] Lustre: server umount lustre-MDT0000 complete [ 675.296798] LustreError: 23253:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 675.323497] LustreError: 23253:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 8 previous similar messages [ 680.415795] Lustre: 3637:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787847810/real 1787847810] req@ffff8bc786e2e1c0 x1874693690944128/t0(0) o400->MGC192.168.204.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787847826 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 680.463908] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 685.146947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 691.680228] LustreError: 3635:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8bc68892a580 x1874693690946816/t0(0) o250->MGC192.168.204.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 692.158412] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 697.320812] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 697.346797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 697.355957] Lustre: Skipped 1 previous similar message [ 697.390491] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 697.536896] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 708.858602] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 714.730069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 714.759464] Lustre: Skipped 5 previous similar messages [ 719.274234] Lustre: server umount lustre-MDT0000 complete [ 723.709643] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847869 with bad export cookie 12659184985676629802 [ 723.717691] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 723.726689] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 724.128769] Lustre: server umount lustre-MDT0001 complete [ 734.511194] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 12:24:37 (1787847877) [ 751.672654] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 762.838787] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 803.618921] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 815.310800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 815.586717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 815.621428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 815.697462] Lustre: lustre-MDT0000: new disk, initializing [ 815.861715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 815.869528] Lustre: Skipped 2 previous similar messages [ 815.905119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 820.243574] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 830.407606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 830.552856] Lustre: 29632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 830.828073] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 835.952223] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 847.881967] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 854.269059] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 861.930218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 862.319509] Lustre: 31603:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 863.668951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 863.717130] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 863.740132] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 863.747552] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 868.053702] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 877.988428] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 884.185371] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 909.795804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 909.798932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 909.816096] Lustre: Skipped 3 previous similar messages [ 911.877763] Lustre: server umount lustre-OST0000 complete [ 917.984474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 922.738664] Lustre: server umount lustre-MDT0000 complete [ 925.153828] LustreError: 29645:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 925.171730] LustreError: 29645:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 13 previous similar messages [ 926.571904] LustreError: 29624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848072 with bad export cookie 12659184985676630208 [ 926.589529] LustreError: 29624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 926.592996] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 927.081770] Lustre: server umount lustre-MDT0001 complete [ 937.959540] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 940.606222] Key type lgssc unregistered [ 940.917347] LNet: 33583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.924821] LNetError: 33583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.939847] LNet: Removed LNI 192.168.204.109@tcp [ 941.915182] Key type .llcrypt unregistered [ 941.918603] Key type ._llcrypt unregistered [ 970.832860] Key type ._llcrypt registered [ 970.836620] Key type .llcrypt registered [ 970.909220] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 971.835220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 971.854200] alg: No test for adler32 (adler32-zlib) [ 972.888316] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 973.084050] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 974.727352] Key type lgssc registered [ 976.382349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 983.723284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 983.746270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 985.114499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.162926] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 995.300645] LustreError: 34996:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 995.322869] LustreError: 34996:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 999.514251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 999.913631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1004.922794] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1013.848981] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1020.443392] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1027.946683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1028.254235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1030.329346] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1035.448622] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1043.911604] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1044.661474] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1058.620564] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 12:30:02 (1787848202) [ 1061.017191] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1063.524537] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1071.123851] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 12:30:14 (1787848214) [ 1081.863101] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 12:30:25 (1787848225) [ 1083.050836] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1093.644952] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1093.649820] Lustre: Skipped 2 previous similar messages [ 1096.235686] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 12:30:39 (1787848239) [ 1098.841952] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1098.848401] Lustre: Skipped 2 previous similar messages [ 1108.263987] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1108.281091] Lustre: Skipped 4 previous similar messages [ 1124.847856] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 12:31:08 (1787848268) [ 1127.930309] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1132.164097] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1142.332819] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 12:31:25 (1787848285) [ 1144.700082] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1144.705833] Lustre: Skipped 6 previous similar messages [ 1160.542615] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 12:31:44 (1787848304) [ 1178.087841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1178.096553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1178.110457] Lustre: Skipped 1 previous similar message [ 1183.226154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1183.235159] Lustre: Skipped 2 previous similar messages [ 1188.334862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1188.338749] Lustre: Skipped 1 previous similar message [ 1189.345331] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1189.559874] Lustre: server umount lustre-OST0000 complete [ 1194.464962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1194.471764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1194.482663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1198.562715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1198.563480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1198.601754] Lustre: Skipped 1 previous similar message [ 1199.995327] Lustre: server umount lustre-MDT0000 complete [ 1203.682407] LustreError: 34996:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1203.698662] LustreError: 34996:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 1204.121874] LustreError: 34976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848349 with bad export cookie 3579021024044526345 [ 1204.126517] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1204.131550] LustreError: 34976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1204.505868] Lustre: server umount lustre-MDT0001 complete [ 1214.652780] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 1217.319144] Key type lgssc unregistered [ 1217.611431] LNet: 42862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1217.625512] LNetError: 42862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1217.665483] LNet: Removed LNI 192.168.204.109@tcp [ 1218.538077] Key type .llcrypt unregistered [ 1218.539943] Key type ._llcrypt unregistered [ 1240.452337] Key type ._llcrypt registered [ 1240.456751] Key type .llcrypt registered [ 1240.595860] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 1258.260040] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 1259.958433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1260.006714] alg: No test for adler32 (adler32-zlib) [ 1260.961511] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 1261.233845] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 1262.895301] Key type lgssc registered [ 1263.893491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1300.217595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1300.247043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1301.526346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1301.556214] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1301.622724] Lustre: lustre-MDT0000: new disk, initializing [ 1301.691147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1301.712177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1306.304920] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1317.919360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1318.043598] Lustre: 46015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1318.095797] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1318.100119] Lustre: Skipped 1 previous similar message [ 1318.183690] Lustre: lustre-MDT0001: new disk, initializing [ 1318.255206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1318.296440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1318.314400] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1323.269399] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1333.246545] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1338.855527] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1345.907545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1346.275156] Lustre: lustre-OST0000: new disk, initializing [ 1346.284263] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1346.292256] Lustre: 47987:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1346.375879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.042291] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1356.382375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1356.396560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1356.578164] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1363.755803] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1369.419742] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1369.660410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1374.890947] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1375.059277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1377.248772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1377.251962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1377.265113] LustreError: Skipped 1 previous similar message [ 1377.272485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1377.292583] Lustre: Skipped 1 previous similar message [ 1382.372491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1382.383583] Lustre: Skipped 2 previous similar messages [ 1387.490263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1387.506220] Lustre: Skipped 1 previous similar message [ 1391.071138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1391.302330] Lustre: server umount lustre-OST0000 complete [ 1397.733320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.748938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1397.754518] Lustre: Skipped 1 previous similar message [ 1397.775370] Lustre: Skipped 1 previous similar message [ 1402.008300] Lustre: server umount lustre-MDT0000 complete [ 1406.387091] LustreError: 46009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848551 with bad export cookie 4612258794522087645 [ 1406.388425] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1406.402693] LustreError: 46009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1406.888938] Lustre: server umount lustre-MDT0001 complete [ 1418.425765] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 12:36:02 (1787848562) [ 1426.043158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1426.624489] LustreError: 50447:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.651205] LustreError: 50447:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 1426.741480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.091302] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1432.042637] LustreError: 50448:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1437.156842] LustreError: 50447:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1440.113571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1440.422147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1445.368569] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1453.222968] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1458.536548] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1465.509428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1465.877715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1471.946715] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1480.906802] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1490.516096] LustreError: 52977:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1491.614952] Lustre: Modifying parameter general.timeout=1 in log params [ 1492.727806] Lustre: Modifying parameter general.timeout=2 in log params [ 1495.613552] Lustre: Modifying parameter general.timeout=5 in log params [ 1495.623202] Lustre: Skipped 2 previous similar messages [ 1499.959436] Lustre: Modifying parameter general.timeout=9 in log params [ 1499.963248] Lustre: Skipped 3 previous similar messages [ 1508.762407] Lustre: Modifying parameter general.timeout=18 in log params [ 1508.767677] Lustre: Skipped 8 previous similar messages [ 1525.377964] Lustre: Modifying parameter general.timeout=32 in log params [ 1525.387375] Lustre: Skipped 13 previous similar messages [ 1558.022553] Lustre: Modifying parameter general.timeout=65 in log params [ 1558.027848] Lustre: Skipped 32 previous similar messages [ 1625.424434] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 12:39:29 (1787848769) [ 1626.916511] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1626.927501] Lustre: Skipped 65 previous similar messages [ 1629.560651] Lustre: 59924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1632.565367] Lustre: 60023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1635.405560] Lustre: 60138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1639.928727] Lustre: 60335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1639.935623] Lustre: 60335:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1673.747556] Lustre: 61807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1673.757409] Lustre: 61807:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1673.778056] LustreError: 61807:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1677.935633] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 12:40:21 (1787848821) [ 1696.736099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1696.746469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1696.753069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1696.773501] Lustre: Skipped 2 previous similar messages [ 1696.781437] Lustre: Skipped 1 previous similar message [ 1698.141770] Lustre: server umount lustre-MDT0000 complete [ 1701.724807] LustreError: 50429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848847 with bad export cookie 4612258794522088527 [ 1701.726751] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1701.735234] LustreError: 50429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1701.856397] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1701.857021] LustreError: 50444:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1702.114267] Lustre: server umount lustre-MDT0001 complete [ 1708.488777] Lustre: server umount lustre-OST0000 complete [ 1739.319176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.648214] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1739.921376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1744.687483] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1753.993820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1754.121076] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1754.148771] Lustre: 64340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1759.990562] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1768.289993] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1775.567924] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1783.077639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1783.293493] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1783.590103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1783.595594] Lustre: Skipped 1 previous similar message [ 1790.735401] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1798.016764] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1799.563425] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1799.572363] Lustre: Skipped 22 previous similar messages [ 1799.627570] Lustre: 66907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1807.848572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1807.863276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1807.880694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1807.895295] Lustre: Skipped 2 previous similar messages [ 1813.423499] Lustre: server umount lustre-OST0000 complete [ 1818.594605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1818.606988] Lustre: Skipped 2 previous similar messages [ 1824.742057] Lustre: server umount lustre-MDT0000 complete [ 1828.487612] LustreError: 64333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848974 with bad export cookie 4612258794522091635 [ 1828.491691] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1828.508097] LustreError: 64333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1828.888090] Lustre: server umount lustre-MDT0001 complete [ 1840.912431] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 1843.458403] Key type lgssc unregistered [ 1843.717689] LNet: 68119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1843.728656] LNetError: 68119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1843.743761] LNet: Removed LNI 192.168.204.109@tcp [ 1844.902363] Key type .llcrypt unregistered [ 1844.904116] Key type ._llcrypt unregistered [ 1869.401530] Key type ._llcrypt registered [ 1869.408954] Key type .llcrypt registered [ 1869.498621] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 1870.640618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1870.721961] alg: No test for adler32 (adler32-zlib) [ 1872.190409] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 1872.675419] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 1874.463265] Key type lgssc registered [ 1876.200371] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1885.529183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1885.555578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1887.084818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.321716] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1896.443242] LustreError: 69535:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1896.460770] LustreError: 69535:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1900.608117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1901.106514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1906.424784] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1915.520441] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1922.093211] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1929.280599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1929.990331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1936.603190] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 1943.685629] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1951.853320] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1961.954437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1961.957917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1961.965319] Lustre: Skipped 1 previous similar message [ 1965.365312] Lustre: server umount lustre-MDT0000 complete [ 1967.078139] LustreError: 71022:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1967.108135] LustreError: 71022:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 1968.892751] LustreError: 69514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849114 with bad export cookie 3476471134287464694 [ 1968.899466] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1968.915154] LustreError: 69514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1969.259591] Lustre: server umount lustre-MDT0001 complete [ 1983.998439] Lustre: server umount lustre-OST0000 complete [ 2012.505109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.854141] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2012.923430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2013.298828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2018.370807] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2028.276743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2028.352686] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2028.412346] Lustre: 74405:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2029.029727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2034.058815] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2042.549581] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2050.115874] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2058.563730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2058.799719] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2059.241081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2066.632051] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2076.779651] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2079.230322] Lustre: 76982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2081.699874] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2088.416031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2088.419543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.434076] LustreError: Skipped 1 previous similar message [ 2088.436468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2088.448213] Lustre: Skipped 2 previous similar messages [ 2088.460647] Lustre: Skipped 3 previous similar messages [ 2093.540782] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2093.560548] Lustre: Skipped 1 previous similar message [ 2094.028727] Lustre: server umount lustre-OST0000 complete [ 2098.672551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.674459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.697912] Lustre: Skipped 1 previous similar message [ 2098.713995] Lustre: Skipped 1 previous similar message [ 2103.776642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2103.788035] Lustre: Skipped 1 previous similar message [ 2104.818906] Lustre: server umount lustre-MDT0000 complete [ 2108.901897] LustreError: 75349:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2108.946023] LustreError: 75349:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2109.306378] LustreError: 76984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849254 with bad export cookie 3476471134287465534 [ 2109.316899] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2109.333310] LustreError: 76984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2109.752620] Lustre: server umount lustre-MDT0001 complete [ 2123.690785] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 2126.717374] Key type lgssc unregistered [ 2127.124069] LNet: 78195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2127.134484] LNetError: 78195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2127.159407] LNet: Removed LNI 192.168.204.109@tcp [ 2128.036337] Key type .llcrypt unregistered [ 2128.046314] Key type ._llcrypt unregistered [ 2160.248818] Key type ._llcrypt registered [ 2160.252219] Key type .llcrypt registered [ 2160.409616] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 2161.979995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2162.004805] alg: No test for adler32 (adler32-zlib) [ 2163.202944] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 2163.575800] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 2165.255259] Key type lgssc registered [ 2166.636801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2176.024383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2176.047438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.688931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2182.570288] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2188.258831] LustreError: 79609:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.278499] LustreError: 79609:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2192.970057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.577631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2199.073418] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2208.569153] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2214.763784] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2222.797574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2223.538143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.163700] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2240.902266] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2252.567264] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2253.555705] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2255.392263] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2255.397154] Lustre: Skipped 1 previous similar message [ 2258.222277] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2258.236573] Lustre: Skipped 2 previous similar messages [ 2263.164280] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2263.169313] Lustre: Skipped 4 previous similar messages [ 2271.938273] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2271.943438] Lustre: Skipped 8 previous similar messages [ 2288.217511] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2288.226219] Lustre: Skipped 11 previous similar messages [ 2320.547562] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2320.551628] Lustre: Skipped 27 previous similar messages [ 2383.260051] Lustre: 87727:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2385.488571] Lustre: Disabling parameter general.jobid_var= in log params [ 2385.504086] Lustre: Skipped 39 previous similar messages [ 2389.204952] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 12:52:11 (1787849531) [ 2393.173679] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2395.863498] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 12:52:19 (1787849539) [ 2401.779552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2401.780978] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2401.791178] LustreError: Skipped 1 previous similar message [ 2401.822174] Lustre: Skipped 1 previous similar message [ 2401.826049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2401.834928] Lustre: Skipped 1 previous similar message [ 2406.418094] Lustre: server umount lustre-OST0000 complete [ 2412.003452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2412.007438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2412.018938] Lustre: Skipped 1 previous similar message [ 2412.028198] Lustre: Skipped 1 previous similar message [ 2415.009834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2415.020951] Lustre: Skipped 1 previous similar message [ 2416.525916] Lustre: server umount lustre-MDT0000 complete [ 2420.196988] LustreError: 81917:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2420.222973] LustreError: 81917:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2420.957656] LustreError: 82200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849566 with bad export cookie 13703264101332784731 [ 2420.974858] LustreError: 82200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2420.975759] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2421.430641] Lustre: server umount lustre-MDT0001 complete [ 2432.943532] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 2435.232498] Key type lgssc unregistered [ 2435.529432] LNet: 88980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2435.536858] LNetError: 88980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2435.553169] LNet: Removed LNI 192.168.204.109@tcp [ 2436.251126] Key type .llcrypt unregistered [ 2436.254955] Key type ._llcrypt unregistered [ 2440.489859] Key type ._llcrypt registered [ 2440.491621] Key type .llcrypt registered [ 2440.624874] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2446.118258] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules [ 2447.237229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2447.268537] alg: No test for adler32 (adler32-zlib) [ 2448.303582] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 2448.622961] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 2450.335512] Key type lgssc registered [ 2451.523611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2452.658685] LustreError: 90191:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2468.963759] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 2474.110115] LustreError: 90191:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2478.840303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2478.882147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2480.582835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2485.651855] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2490.850143] LustreError: 91205:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2490.865914] LustreError: 91205:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2495.972729] LustreError: 91206:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.246373] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 12:54:00 (1787849640) [ 2501.090921] LustreError: 91205:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.268382] LustreError: 91237:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2501.288155] LustreError: 91237:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2501.296857] LustreError: 91237:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2501.840992] Lustre: server umount lustre-MDT0000 complete [ 2518.534682] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 2521.666253] Key type lgssc unregistered [ 2521.981510] LNet: 92736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2521.992395] LNetError: 92736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2522.006928] LNet: Removed LNI 192.168.204.109@tcp [ 2522.879860] Key type .llcrypt unregistered [ 2522.885303] Key type ._llcrypt unregistered [ 2543.733628] Key type ._llcrypt registered [ 2543.735961] Key type .llcrypt registered [ 2543.847285] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 2545.290204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2545.308864] alg: No test for adler32 (adler32-zlib) [ 2546.481971] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 2546.761811] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 2548.464135] Key type lgssc registered [ 2549.802689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2558.181763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2558.194647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.707949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2565.258544] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2570.210880] LustreError: 93956:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.234166] LustreError: 93956:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2575.332807] LustreError: 93955:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.813945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2576.671141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2581.277905] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2589.697623] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2597.013819] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2604.191282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2604.780096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.259591] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2618.456274] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2679.513784] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 12:57:03 (1787849823) [ 2681.535718] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2683.645826] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 12:57:07 (1787849827) [ 2689.008867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2689.022649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2689.029931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2693.090147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2693.090346] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2693.125654] Lustre: Skipped 1 previous similar message [ 2694.601814] Lustre: server umount lustre-MDT0000 complete [ 2698.216822] LustreError: 95076:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2698.228091] LustreError: 95076:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 2698.849066] LustreError: 93935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849844 with bad export cookie 11935004270638820868 [ 2698.861581] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2699.124713] Lustre: server umount lustre-MDT0001 complete [ 2714.538065] Lustre: server umount lustre-OST0000 complete [ 2727.412796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2727.920888] LustreError: 98250:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2728.091470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2733.548664] LustreError: 98250:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2733.576852] LustreError: 98250:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2733.661640] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2742.755356] LustreError: 98251:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.767538] LustreError: 98251:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2743.503873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.426336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2749.644233] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2757.545747] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2764.238081] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2780.200448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2780.485022] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2780.499890] LustreError: 98238:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2780.519703] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2780.525837] LustreError: 100335:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2780.532443] LustreError: 100335:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2780.537607] LustreError: 100335:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2780.608993] Lustre: server umount lustre-OST0000 complete [ 2780.612364] LustreError: 100335:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2789.995961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2790.203874] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2790.211596] LustreError: 98238:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2790.236838] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2790.249724] LustreError: 100801:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2790.255403] LustreError: 100801:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2790.259984] LustreError: 100801:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2790.331497] Lustre: server umount lustre-OST0000 complete [ 2790.334880] LustreError: 100801:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2799.449556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2799.766319] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2799.771793] Lustre: Found index 0 for lustre-OST0000, updating log [ 2799.794379] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2799.969697] Lustre: lustre-OST0000: new disk, initializing [ 2799.976666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2799.991458] Lustre: 101333:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2800.277678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.615864] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2810.474220] Lustre: Failing over lustre-OST0000 [ 2810.633305] Lustre: server umount lustre-OST0000 complete [ 2815.462731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2815.475983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.498661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2815.504462] Lustre: Skipped 2 previous similar messages [ 2819.046947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2819.055175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2819.079652] Lustre: Skipped 1 previous similar message [ 2820.569713] Lustre: server umount lustre-MDT0000 complete [ 2824.170638] LustreError: 98250:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2824.193578] LustreError: 98250:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 2824.687104] LustreError: 100354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849970 with bad export cookie 11935004270638823612 [ 2824.689131] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2824.714419] LustreError: 100354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2825.158699] Lustre: server umount lustre-MDT0001 complete [ 2835.959307] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 12:59:39 (1787849979) [ 2855.550404] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 2865.500436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2865.917700] LustreError: 103678:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2866.040350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2871.069584] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2879.748493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2885.122430] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2888.492401] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2896.638895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2904.407772] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2914.035342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2914.170389] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2914.317816] Lustre: lustre-OST0001: new disk, initializing [ 2914.325682] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2914.334608] Lustre: 105999:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2914.662145] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2914.669386] Lustre: Skipped 2 previous similar messages [ 2922.283749] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2924.669348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2924.680741] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2924.767303] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2930.092069] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2937.545591] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2941.928751] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2941.943066] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2941.977254] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2950.638906] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2950.651279] Lustre: Skipped 6 previous similar messages [ 2956.255593] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2956.572256] Lustre: server umount lustre-MDT0001 complete [ 2960.875575] LustreError: 103674:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2960.910878] LustreError: 103674:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [ 2967.121202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2967.256774] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2967.264920] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2967.319939] Lustre: 103666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2969.159577] LustreError: 107580:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2969.166774] LustreError: 107580:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2969.187596] LustreError: 103670:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2969.376950] LustreError: 107597:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.109@tcp: cfg command failed: rc = -17 [ 2969.390699] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2969.390699] [ 2973.167929] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2973.373953] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 2984.742958] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 13:02:08 (1787850128) [ 3004.927659] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3024.360553] Lustre: 109629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3055.459954] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3060.321872] Lustre: 110765:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3157.482589] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3157.484822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3157.491209] Lustre: Skipped 5 previous similar messages [ 3157.494574] Lustre: Skipped 7 previous similar messages [ 3163.591700] Lustre: server umount lustre-MDT0000 complete [ 3167.722230] LustreError: 104692:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3167.772537] LustreError: 104692:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 3168.047678] LustreError: 106731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850313 with bad export cookie 11935004270638824536 [ 3168.055164] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3168.068757] LustreError: 106731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3168.422112] Lustre: server umount lustre-MDT0001 complete [ 3183.745356] Lustre: server umount lustre-OST0000 complete [ 3198.502316] Lustre: server umount lustre-OST0001 complete [ 3206.113063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3217.407965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3226.707358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3238.484065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3262.515470] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3273.259336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3273.285553] Lustre: lustre-MDT0000: reset Object Index mappings [ 3273.868419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3273.872571] Lustre: Skipped 1 previous similar message [ 3279.106498] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3288.823946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3288.854308] Lustre: lustre-MDT0001: reset Object Index mappings [ 3294.483618] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3298.048630] Lustre: 115870:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3306.357625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3313.532651] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3314.729719] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3318.656529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3324.704676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3330.567532] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3330.579771] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3332.503616] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3341.220987] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3345.385616] Lustre: 117750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3403.561676] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 13:09:07 (1787850547) [ 3432.429381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3432.435500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3432.475286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3432.488235] Lustre: Skipped 5 previous similar messages [ 3435.761081] Lustre: server umount lustre-MDT0000 complete [ 3438.053954] LustreError: 114722:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3438.068723] LustreError: 114722:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 14 previous similar messages [ 3439.681733] LustreError: 114703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850585 with bad export cookie 11935004270638871401 [ 3439.690432] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3439.692935] LustreError: 114703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3446.438655] Lustre: server umount lustre-MDT0001 complete [ 3456.953601] Lustre: server umount lustre-OST0000 complete [ 3466.736667] Lustre: server umount lustre-OST0001 complete [ 3474.029883] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 3484.673456] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3526.779427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3527.080872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3527.127338] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3527.221651] Lustre: lustre-MDT0000: new disk, initializing [ 3527.383264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3527.395809] Lustre: Skipped 3 previous similar messages [ 3527.418902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3532.621712] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3541.198063] Lustre: server umount lustre-MDT0000 complete [ 3553.178953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3553.602837] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3553.621210] Lustre: 122023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3553.641686] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3560.399934] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3566.988188] Lustre: server umount lustre-MDT0000 complete [ 3579.051470] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 13:12:02 (1787850722) [ 3581.075756] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3583.607705] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 13:12:07 (1787850727) [ 3606.277969] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 3618.783627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3624.345996] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3634.205117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3634.329275] Lustre: 124074:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3634.375809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3634.379568] Lustre: Skipped 1 previous similar message [ 3634.435406] Lustre: lustre-MDT0001: new disk, initializing [ 3634.541197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3634.552289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3639.931253] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3645.809158] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3654.483121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3654.807414] Lustre: lustre-OST0000: new disk, initializing [ 3654.811714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3654.818348] Lustre: 125656:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3656.924049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3656.933994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3657.111285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3662.940955] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3676.229675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3676.345689] Lustre: lustre-OST0001: new disk, initializing [ 3676.348959] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3676.355817] Lustre: 126526:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3677.589280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3677.623367] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3677.818533] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3689.148938] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 3705.469511] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3711.985734] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4168.352259] Lustre: 132953:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4168.368951] Lustre: 132953:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4168.386065] Lustre: 132953:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4168.395369] Lustre: 132953:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4168.408779] Lustre: 132953:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4168.421037] Lustre: 132953:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4491.470763] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 13:27:13 (1787851633) [ 4508.296576] Lustre: server umount lustre-MDT0000 complete [ 4510.177041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4510.184700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4510.193571] Lustre: Skipped 5 previous similar messages [ 4510.197726] LustreError: 127346:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4510.215901] LustreError: 127346:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [ 4512.787799] LustreError: 124069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851658 with bad export cookie 11935004270638941429 [ 4512.805028] LustreError: 124069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4512.805861] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4517.345252] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4517.375256] Lustre: Skipped 3 previous similar messages [ 4517.388702] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4517.398714] Lustre: Skipped 8 previous similar messages [ 4519.508270] Lustre: server umount lustre-MDT0001 complete [ 4530.342577] Lustre: server umount lustre-OST0000 complete [ 4550.111326] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4550.787317] Lustre: server umount lustre-OST0001 complete [ 4558.865750] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 4573.192280] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 4619.706035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4619.974814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4620.026503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4620.117151] Lustre: lustre-MDT0000: new disk, initializing [ 4620.237178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4620.243911] Lustre: Skipped 5 previous similar messages [ 4620.258543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4625.651731] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4637.011563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4637.219656] Lustre: 136027:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4637.294342] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4637.298790] Lustre: Skipped 1 previous similar message [ 4637.388243] Lustre: lustre-MDT0001: new disk, initializing [ 4637.574952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4637.592313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4643.147592] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4654.657498] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4661.942852] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4670.884130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4671.227803] Lustre: lustre-OST0000: new disk, initializing [ 4671.245209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4671.251201] Lustre: 138009:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4671.374635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4671.378988] Lustre: Skipped 1 previous similar message [ 4673.091848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4673.114134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4673.265985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4678.632424] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 4691.698231] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4703.829848] Lustre: lustre-MDD0000: changelog on [ 4707.113810] Lustre: lustre-MDD0001: changelog on [ 7127.233852] Lustre: lustre-MDD0001: changelog off [ 7129.430821] Lustre: 137703:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 7130.429749] Lustre: lustre-MDD0000: changelog off [ 7136.227086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7136.233438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7136.235027] LustreError: Skipped 1 previous similar message [ 7136.239401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7136.253596] Lustre: Skipped 2 previous similar messages [ 7136.267481] Lustre: Skipped 1 previous similar message [ 7139.303592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7139.309687] Lustre: Skipped 1 previous similar message [ 7142.466501] Lustre: server umount lustre-OST0000 complete [ 7149.538487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7149.546882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7149.549820] Lustre: Skipped 1 previous similar message [ 7152.912751] Lustre: server umount lustre-MDT0000 complete [ 7154.660033] LustreError: 136039:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7154.673501] LustreError: 136039:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 7156.733791] LustreError: 136021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854302 with bad export cookie 11935004270639722237 [ 7156.735191] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7156.763884] LustreError: 136021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7157.101632] Lustre: server umount lustre-MDT0001 complete [ 7169.568497] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 7172.506991] Key type lgssc unregistered [ 7172.803807] LNet: 147242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7172.810974] LNetError: 147242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7173.861880] LNet: Removed LNI 192.168.204.109@tcp [ 7174.851174] Key type .llcrypt unregistered [ 7174.852764] Key type ._llcrypt unregistered [ 7195.937943] Key type ._llcrypt registered [ 7195.941201] Key type .llcrypt registered [ 7196.108569] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 7214.409443] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 7215.880845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7215.960973] alg: No test for adler32 (adler32-zlib) [ 7217.067789] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 7217.334874] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 7219.023217] Key type lgssc registered [ 7220.276798] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7256.497239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7256.538447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7257.744565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7257.767450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7257.854705] Lustre: lustre-MDT0000: new disk, initializing [ 7257.923656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7257.934449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7262.021595] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7275.448660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7275.644913] Lustre: 150445:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7275.701136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7275.705470] Lustre: Skipped 1 previous similar message [ 7275.816810] Lustre: lustre-MDT0001: new disk, initializing [ 7275.923292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7275.967495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7275.977654] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7282.873850] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7293.273866] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7299.279502] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7306.815379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7307.074784] Lustre: lustre-OST0000: new disk, initializing [ 7307.078845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7307.086995] Lustre: 152415:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7307.165951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7314.575647] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7315.060304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7315.068614] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7315.192420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7325.726214] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7332.279129] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_module obdecho/obdecho [ 7334.148976] LustreError: 153418:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7334.173626] LustreError: 153418:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7336.054683] LustreError: 153515:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7351.266454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7351.275196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7351.285487] Lustre: Skipped 2 previous similar messages [ 7351.299484] Lustre: Skipped 2 previous similar messages [ 7354.392871] Lustre: server umount lustre-MDT0000 complete [ 7358.021712] LustreError: 152414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854503 with bad export cookie 12598768301115914340 [ 7358.039196] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7358.039897] LustreError: 152414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7358.445446] Lustre: server umount lustre-MDT0001 complete [ 7372.553876] Lustre: server umount lustre-OST0000 complete [ 7397.022596] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 7408.230539] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 7452.884236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7453.192550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7453.237270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7453.393330] Lustre: lustre-MDT0000: new disk, initializing [ 7453.509950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7453.530964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7458.092408] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7469.429072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7469.672139] Lustre: 157234:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7469.803270] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7469.809026] Lustre: Skipped 1 previous similar message [ 7469.971374] Lustre: lustre-MDT0001: new disk, initializing [ 7470.122523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7470.167128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7470.177914] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7475.250027] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7485.897817] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7492.299130] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7499.896115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7500.126917] Lustre: lustre-OST0000: new disk, initializing [ 7500.131038] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7500.136878] Lustre: 159200:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7500.224299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7506.506629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7506.524291] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7506.660645] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7507.104429] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7518.350653] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7523.422362] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7523.623283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7527.984619] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7528.225495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7532.517467] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7532.519032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7532.531556] LustreError: Skipped 1 previous similar message [ 7532.534030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7532.561364] Lustre: Skipped 1 previous similar message [ 7536.535985] Lustre: server umount lustre-OST0000 complete [ 7541.729135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7541.736422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7541.754707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7541.768420] Lustre: Skipped 1 previous similar message [ 7546.181431] Lustre: server umount lustre-MDT0000 complete [ 7547.872727] LustreError: 157246:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7547.891234] LustreError: 157246:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [ 7550.118192] LustreError: 157225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854695 with bad export cookie 12598768301115915299 [ 7550.127802] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7550.131280] LustreError: 157225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7550.514777] Lustre: server umount lustre-MDT0001 complete [ 7554.272699] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 14:18:18 (1787854698) [ 7556.245471] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7558.256906] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 14:18:22 (1787854702) [ 7566.389466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7566.756140] LustreError: 161613:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7566.847185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7571.835661] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7571.957494] LustreError: 161614:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7577.066358] LustreError: 161613:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7582.180247] LustreError: 161614:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7582.394962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7589.453519] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7597.496511] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7604.061193] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7611.334476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7611.636759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7611.643394] Lustre: Skipped 1 previous similar message [ 7619.304313] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7627.149094] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7638.060620] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.9@tcp (stopping) [ 7638.070113] Lustre: Skipped 2 previous similar messages [ 7639.266593] Lustre: server umount lustre-MDT0000 complete [ 7639.521180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7639.534068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7639.546823] Lustre: Skipped 1 previous similar message [ 7639.558644] LustreError: 161614:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7642.776754] LustreError: 161595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854788 with bad export cookie 12598768301115916181 [ 7642.789214] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7642.798926] LustreError: 161595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7643.173263] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.9@tcp (stopping) [ 7646.690724] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7646.702241] Lustre: Skipped 2 previous similar messages [ 7649.194971] Lustre: server umount lustre-MDT0001 complete [ 7656.286696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7670.461700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7670.706157] LustreError: 165456:0:(llog.c:1655:llog_backup()) MGC192.168.204.109@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7670.714828] Lustre: 165456:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.109@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7673.316570] LustreError: 165461:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7673.333339] LustreError: 165461:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 7 previous similar messages [ 7708.130479] LustreError: 165463:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7708.147442] LustreError: 165463:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 27 previous similar messages [ 7731.679461] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7731.689654] Lustre: 165456:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.109@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7772.667254] LustreError: 165479:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7772.668315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7772.687878] LustreError: 165479:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 44 previous similar messages [ 7777.773619] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7777.898455] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7787.102335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7793.440725] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7793.636808] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7801.654477] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7803.254273] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7809.525930] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7811.281148] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7833.368207] Lustre: server umount lustre-MDT0000 complete [ 7834.594762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7834.608037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7834.615911] Lustre: Skipped 1 previous similar message [ 7837.187195] LustreError: 161596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787854982 with bad export cookie 12598768301115922635 [ 7837.200442] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7837.201015] LustreError: 161596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7837.537477] Lustre: server umount lustre-MDT0001 complete [ 7852.057163] Lustre: server umount lustre-OST0000 complete [ 7876.312414] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 7888.416399] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 7930.610157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7930.957650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7931.000476] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7931.062829] Lustre: lustre-MDT0000: new disk, initializing [ 7931.216491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7931.221152] Lustre: Skipped 1 previous similar message [ 7931.250844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7936.221486] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7947.746714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7947.939581] Lustre: 170561:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7948.230502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7954.212561] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7963.741639] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7970.367755] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7976.952154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7977.279371] Lustre: lustre-OST0000: new disk, initializing [ 7977.282320] Lustre: Skipped 1 previous similar message [ 7977.287431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7977.302508] Lustre: Skipped 2 previous similar messages [ 7977.324374] Lustre: 172532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7983.641605] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 7986.254311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7986.276463] Lustre: Skipped 1 previous similar message [ 7986.284634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7986.400858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7993.080169] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7997.050866] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7997.261888] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8001.848321] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8002.172441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8007.138224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8007.148831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8007.179680] Lustre: Skipped 1 previous similar message [ 8007.202843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8007.215963] Lustre: Skipped 3 previous similar messages [ 8010.668800] Lustre: server umount lustre-OST0000 complete [ 8022.165991] Lustre: server umount lustre-MDT0000 complete [ 8022.501513] LustreError: 170572:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8022.531614] LustreError: 170572:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 15 previous similar messages [ 8026.354020] LustreError: 170555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855171 with bad export cookie 12598768301115939463 [ 8026.362833] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8026.374709] LustreError: 170555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8026.713131] Lustre: server umount lustre-MDT0001 complete [ 8037.318670] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 14:26:21 (1787855181) [ 8044.168299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8049.574572] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8058.302358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8062.587775] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8070.226522] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8077.635723] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8086.462155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8094.007834] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8101.395415] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8115.967958] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 14:27:39 (1787855259) [ 8120.801491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8120.814168] LustreError: Skipped 1 previous similar message [ 8120.828271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8120.844779] Lustre: Skipped 3 previous similar messages [ 8120.854767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8120.864465] Lustre: Skipped 3 previous similar messages [ 8127.152563] Lustre: server umount lustre-OST0000 complete [ 8145.376875] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8145.592708] Lustre: server umount lustre-MDT0000 complete [ 8149.130928] LustreError: 174924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855294 with bad export cookie 12598768301115940352 [ 8149.134387] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8149.139618] LustreError: 174924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8149.416518] Lustre: server umount lustre-MDT0001 complete [ 8159.741578] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 8162.560455] Key type lgssc unregistered [ 8162.815966] LNet: 178678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8162.822948] LNetError: 178678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8162.841530] LNet: Removed LNI 192.168.204.109@tcp [ 8163.644726] Key type .llcrypt unregistered [ 8163.647171] Key type ._llcrypt unregistered [ 8182.790249] Key type ._llcrypt registered [ 8182.793927] Key type .llcrypt registered [ 8182.932732] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 8184.349868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8184.414496] alg: No test for adler32 (adler32-zlib) [ 8185.455134] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 8185.643992] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 8187.327188] Key type lgssc registered [ 8188.453885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8195.963841] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8195.979564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8211.553705] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8216.677719] LustreError: 179925:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.109@tcp: failed processing log, type 4: rc = -110 [ 8221.792134] LustreError: 179925:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8221.818964] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8221.830260] LustreError: MGC192.168.204.109@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8221.854351] LustreError: 179925:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8221.876705] LustreError: 179925:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8221.891362] LustreError: 179925:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8222.001173] Lustre: server umount lustre-OST0000 complete [ 8222.012239] LustreError: 179925:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8231.443722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8233.029336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8238.659559] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8243.178030] LustreError: 180425:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8243.201841] LustreError: 180425:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8248.306649] LustreError: 180424:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8249.645894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8250.320954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8256.345524] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8264.949220] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8272.376902] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8280.121048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8280.356635] LustreError: 182241:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8280.379772] Lustre: 182241:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.109@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8280.575709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8287.096920] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8295.414939] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8300.003852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8300.011276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8300.013701] Lustre: Skipped 2 previous similar messages [ 8300.022773] Lustre: Skipped 2 previous similar messages [ 8303.981655] Lustre: server umount lustre-MDT0000 complete [ 8305.127813] LustreError: 180419:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8305.159710] LustreError: 180419:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8307.822372] LustreError: 180406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855453 with bad export cookie 15262644863207080799 [ 8307.825262] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8307.834678] LustreError: 180406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8308.155534] Lustre: server umount lustre-MDT0001 complete [ 8322.281017] Lustre: server umount lustre-OST0000 complete [ 8332.809495] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 8345.085615] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 8387.340171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8387.687348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8387.740119] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8387.846685] Lustre: lustre-MDT0000: new disk, initializing [ 8387.901782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8387.925873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8392.263613] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8405.502766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8405.782583] Lustre: 185767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8405.893968] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8405.901814] Lustre: Skipped 1 previous similar message [ 8406.065938] Lustre: lustre-MDT0001: new disk, initializing [ 8406.206515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8406.289440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8406.315125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8412.382056] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8424.887373] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8431.121033] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8439.292836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8439.518569] Lustre: lustre-OST0000: new disk, initializing [ 8439.526640] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8439.537842] Lustre: 187734:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8439.637947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8446.392569] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8448.098799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8448.114827] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8448.247222] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8456.946824] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8463.087474] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8463.333508] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8469.667660] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8469.966477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8474.084926] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8474.090267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8474.107776] LustreError: Skipped 1 previous similar message [ 8474.112596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8474.154076] Lustre: Skipped 1 previous similar message [ 8479.145368] Lustre: server umount lustre-OST0000 complete [ 8488.419881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8488.431063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8488.456184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8488.463024] Lustre: Skipped 1 previous similar message [ 8491.067891] Lustre: server umount lustre-MDT0000 complete [ 8493.552854] LustreError: 187136:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8493.584579] LustreError: 187136:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8496.519210] LustreError: 185761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855642 with bad export cookie 15262644863207081513 [ 8496.522261] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8496.548111] LustreError: 185761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8496.822201] Lustre: server umount lustre-MDT0001 complete [ 8509.884683] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 14:34:13 (1787855653) [ 8530.759064] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 8533.914854] Key type lgssc unregistered [ 8534.290058] LNet: 190687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8534.296913] LNetError: 190687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8534.318052] LNet: Removed LNI 192.168.204.109@tcp [ 8535.574087] Key type .llcrypt unregistered [ 8535.580308] Key type ._llcrypt unregistered [ 8556.979480] Key type ._llcrypt registered [ 8556.984329] Key type .llcrypt registered [ 8557.124217] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 8558.228926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8558.249017] alg: No test for adler32 (adler32-zlib) [ 8559.532661] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 8559.881922] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 8561.623224] Key type lgssc registered [ 8563.165885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8572.951131] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8572.974181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8574.610785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8579.772145] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8585.191071] LustreError: 191903:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8585.217238] LustreError: 191903:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8590.307627] LustreError: 191904:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8590.801145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8591.378426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8596.746965] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8605.572994] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8613.365272] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8623.973626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8624.354306] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8624.358168] LustreError: 193770:0:(llog.c:1696:llog_backup()) MGC192.168.204.109@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8624.367148] Lustre: 193770:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.109@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8624.546920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8632.641556] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8642.713817] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8647.653669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8647.661185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8647.680725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8650.215526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8650.219415] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8651.937530] Lustre: server umount lustre-MDT0000 complete [ 8655.336325] LustreError: 191904:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8655.360845] LustreError: 191904:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 8656.001967] LustreError: 191883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855801 with bad export cookie 14624660608975184431 [ 8656.010254] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8656.022960] LustreError: 191883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8656.465952] Lustre: server umount lustre-MDT0001 complete [ 8671.787091] Lustre: server umount lustre-OST0000 complete [ 8684.627613] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 8695.083882] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 8738.065438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8738.346220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8738.380096] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8738.489087] Lustre: lustre-MDT0000: new disk, initializing [ 8738.582365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8738.612881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8742.591396] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8754.647540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8754.725599] Lustre: 197297:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8754.758694] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8754.760591] Lustre: Skipped 1 previous similar message [ 8754.766268] LustreError: 198078:0:(llog.c:1696:llog_backup()) MGC192.168.204.109@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8754.771755] LustreError: 198078:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8754.775773] Lustre: 198078:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.109@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8754.788604] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8754.795542] Lustre: Skipped 1 previous similar message [ 8754.869845] Lustre: lustre-MDT0001: new disk, initializing [ 8754.957171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8755.009675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8755.031165] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8759.850939] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8771.812913] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8780.096965] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8789.694801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8790.136752] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8790.142459] Lustre: Skipped 1 previous similar message [ 8790.144603] LustreError: 199241:0:(llog.c:1696:llog_backup()) MGC192.168.204.109@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8790.151656] LustreError: 199241:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8790.165236] Lustre: 199241:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.109@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8790.182231] Lustre: 199241:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8790.214342] Lustre: lustre-OST0000: new disk, initializing [ 8790.228715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8790.240917] Lustre: 199265:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8790.376918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8798.378574] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8799.281801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8799.287778] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8799.415907] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8808.774577] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8814.964052] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8815.224671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8821.506976] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8821.782817] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8824.813947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8824.823127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8824.836426] Lustre: Skipped 1 previous similar message [ 8824.849254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8824.856528] Lustre: Skipped 2 previous similar messages [ 8830.243922] Lustre: server umount lustre-OST0000 complete [ 8835.554359] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8835.573970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8835.578392] Lustre: Skipped 2 previous similar messages [ 8835.602516] Lustre: Skipped 3 previous similar messages [ 8840.674904] LustreError: 197307:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8840.752340] Lustre: server umount lustre-MDT0000 complete [ 8844.876277] LustreError: 197289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787855990 with bad export cookie 14624660608975185152 [ 8844.877802] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8844.884666] LustreError: 197289:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8845.587772] Lustre: server umount lustre-MDT0001 complete [ 8857.208993] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 14:40:00 (1787856000) [ 8876.275672] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 8888.974543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8889.423948] LustreError: 202399:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8889.440275] LustreError: 202399:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8889.526613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8895.176531] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8900.069377] LustreError: 202400:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8900.082756] LustreError: 202400:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [ 8905.609573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8911.479462] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8915.278265] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8923.561311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8923.904386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8923.911705] Lustre: Skipped 1 previous similar message [ 8930.299555] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8940.029463] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8940.197401] Lustre: lustre-OST0001: new disk, initializing [ 8940.200480] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8940.205553] Lustre: 204705:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8947.790248] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 8948.864370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8948.878644] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8948.987482] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8959.828473] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8968.632136] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8970.876885] Lustre: DEBUG MARKER: ADD OST3 [ 8977.087911] loop: module loaded [ 8977.159971] loop0: detected capacity change from 0 to 204800000 [ 8977.423482] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8977.431208] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8977.448866] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8985.241992] Lustre: DEBUG MARKER: STOP OST3 [ 8985.538587] loop0: detected capacity change from 0 to 204800000 [ 8992.337024] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8992.619585] Lustre: lustre-OST0003: new disk, initializing [ 8992.626954] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8992.633985] Lustre: 206755:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8992.761519] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8992.767566] Lustre: Skipped 1 previous similar message [ 8994.281130] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8994.436279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8994.446298] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8994.506772] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8995.826652] Lustre: Failing over lustre-OST0003 [ 8996.113537] Lustre: server umount lustre-OST0003 complete [ 8996.321812] LustreError: 206893:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8999.418824] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8999.443934] LustreError: 203888:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8999.462700] LustreError: 203888:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 9000.559896] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9013.567372] Lustre: DEBUG MARKER: START OST3 again [ 9022.133588] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9022.459961] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 9024.182271] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9024.448029] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 9024.449399] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 9028.382238] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9051.322340] Lustre: Failing over lustre-OST0003 [ 9051.480721] Lustre: server umount lustre-OST0003 complete [ 9053.178124] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9053.189835] LustreError: 203885:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9053.198660] Lustre: Skipped 2 previous similar messages [ 9053.224491] LustreError: 203885:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 14 previous similar messages [ 9073.632535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9073.642388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9073.653910] Lustre: Skipped 3 previous similar messages [ 9073.674529] Lustre: Skipped 3 previous similar messages [ 9076.510711] Lustre: server umount lustre-MDT0000 complete [ 9080.955234] LustreError: 202380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856226 with bad export cookie 14624660608975186034 [ 9080.982251] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9080.994165] LustreError: 202380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9083.881624] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9087.423151] Lustre: server umount lustre-MDT0001 complete [ 9092.702323] Lustre: server umount lustre-OST0000 complete [ 9098.016281] Lustre: server umount lustre-OST0001 complete [ 9104.980714] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 9115.004527] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 9160.291222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9160.704899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9160.756522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9160.881851] Lustre: lustre-MDT0000: new disk, initializing [ 9160.973116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9160.981103] Lustre: Skipped 1 previous similar message [ 9160.997870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9166.870930] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9178.532950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9178.652224] Lustre: 212321:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9178.704363] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9178.708702] Lustre: Skipped 1 previous similar message [ 9178.767599] Lustre: lustre-MDT0001: new disk, initializing [ 9178.897058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9178.910881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9184.272309] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9194.464208] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9200.905424] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9208.403760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9208.693371] Lustre: 214291:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9210.640267] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9210.817802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9216.342990] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9227.819445] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9234.272276] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9234.629703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9241.301153] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9241.735069] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9246.695857] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9246.697068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9246.701858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9246.701869] Lustre: Skipped 3 previous similar messages [ 9246.709555] LustreError: Skipped 1 previous similar message [ 9246.739314] Lustre: Skipped 3 previous similar messages [ 9250.437467] Lustre: server umount lustre-OST0000 complete [ 9255.905699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9255.920215] LustreError: Skipped 1 previous similar message [ 9262.060692] LustreError: 212333:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9262.086578] LustreError: 212333:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 17 previous similar messages [ 9262.359885] Lustre: server umount lustre-MDT0000 complete [ 9267.870758] LustreError: 214290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856413 with bad export cookie 14624660608975188330 [ 9267.874856] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9267.892386] LustreError: 214290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9268.312600] Lustre: server umount lustre-MDT0001 complete [ 9273.057442] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 14:46:56 (1787856416) [ 9291.774768] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 9303.238383] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 9346.750537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9347.245048] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9347.299107] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9347.305047] Lustre: Skipped 1 previous similar message [ 9347.423621] Lustre: lustre-MDT0000: new disk, initializing [ 9347.431780] Lustre: Skipped 1 previous similar message [ 9347.595229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9347.621300] Lustre: Skipped 1 previous similar message [ 9353.346599] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9365.144517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.366256] Lustre: 218904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9366.126168] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9373.375150] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9384.395232] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9391.347860] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9398.851723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9399.218699] Lustre: 220874:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9401.158855] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9401.351935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9406.574078] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9415.773671] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9420.690303] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9420.931517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9426.318758] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9426.542802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9432.040396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9432.041138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9432.054279] LustreError: Skipped 1 previous similar message [ 9432.056887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9432.065848] Lustre: Skipped 3 previous similar messages [ 9432.084551] Lustre: Skipped 5 previous similar messages [ 9443.295257] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9443.482599] Lustre: server umount lustre-OST0000 complete [ 9453.778559] Lustre: server umount lustre-MDT0000 complete [ 9457.638071] LustreError: 222214:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9457.651225] LustreError: 222214:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [ 9457.740372] LustreError: 218896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856603 with bad export cookie 14624660608975189212 [ 9457.751692] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9457.752649] LustreError: 218896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9458.111914] Lustre: server umount lustre-MDT0001 complete [ 9465.835866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9466.372762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9466.381222] Lustre: Skipped 5 previous similar messages [ 9470.404549] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9478.941154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9484.615985] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9492.278467] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9498.528095] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9506.678221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9515.097528] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9524.393355] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9534.955578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9534.970911] Lustre: Skipped 9 previous similar messages [ 9540.901398] Lustre: server umount lustre-OST0000 complete [ 9551.255147] Lustre: server umount lustre-MDT0000 complete [ 9555.986375] LustreError: 223073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787856701 with bad export cookie 14624660608975190094 [ 9555.990141] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9556.003429] LustreError: 223073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9556.269969] Lustre: server umount lustre-MDT0001 complete [ 9666.516404] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [ 9669.651024] Key type lgssc unregistered [ 9670.008810] LNet: 226761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9670.015416] LNetError: 226761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9670.030151] LNet: Removed LNI 192.168.204.109@tcp [ 9670.937148] Key type .llcrypt unregistered [ 9670.938744] Key type ._llcrypt unregistered [ 9698.828985] Key type ._llcrypt registered [ 9698.833338] Key type .llcrypt registered [ 9698.958268] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [ 9720.547330] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 9721.654732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9721.722712] alg: No test for adler32 (adler32-zlib) [ 9722.895465] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [ 9723.127792] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [ 9724.799176] Key type lgssc registered [ 9726.206534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9764.119141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9764.177217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9765.414103] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9765.437540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9765.519945] Lustre: lustre-MDT0000: new disk, initializing [ 9765.589718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9765.614467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9770.128857] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9782.902888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9783.155046] Lustre: 230117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9783.252422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9783.259994] Lustre: Skipped 1 previous similar message [ 9783.365825] Lustre: lustre-MDT0001: new disk, initializing [ 9783.545880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9783.628511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9783.640191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9789.231944] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9799.656400] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9806.659360] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9814.530746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9814.949486] Lustre: lustre-OST0000: new disk, initializing [ 9814.954414] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9814.964573] Lustre: 232088:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9815.064942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9820.749318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9820.764870] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9820.895185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9822.783929] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9833.685966] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9839.869549] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9840.117875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9845.301465] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9845.525743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9851.874662] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9851.880716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9851.887279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9851.909296] Lustre: Skipped 1 previous similar message [ 9856.992586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9857.000408] Lustre: Skipped 2 previous similar messages [ 9862.111479] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9862.114885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9862.142942] Lustre: Skipped 1 previous similar message [ 9862.327318] Lustre: server umount lustre-OST0000 complete [ 9867.234611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9867.243545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9867.255424] Lustre: Skipped 1 previous similar message [ 9872.359395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9872.364910] Lustre: Skipped 2 previous similar messages [ 9873.023026] Lustre: server umount lustre-MDT0000 complete [ 9877.243311] LustreError: 230108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857022 with bad export cookie 10981548067847065520 [ 9877.259098] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9877.261300] LustreError: 230108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9877.785613] Lustre: server umount lustre-MDT0001 complete [ 9900.830551] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [ 9911.684950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9912.353615] LustreError: 235028:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9912.376842] LustreError: 235028:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [ 9912.440178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9917.919916] LustreError: 235029:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9918.290691] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9922.025958] LustreError: 235028:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9927.137533] LustreError: 235029:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9928.932804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9929.358270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9934.162636] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9937.460603] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9944.605112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9945.082411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9951.819745] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9960.831584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9961.085791] Lustre: lustre-OST0001: new disk, initializing [ 9961.090757] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9961.107348] Lustre: 237336:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9961.266653] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9967.779431] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [ 9968.786942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9968.798893] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9968.873255] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9979.266151] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9987.055703] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10005.129524] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 14:59:09 (1787857149) [10010.081597] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10010.089244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10010.101686] Lustre: Skipped 3 previous similar messages [10010.122303] Lustre: Skipped 3 previous similar messages [10013.570319] Lustre: server umount lustre-MDT0000 complete [10015.204690] LustreError: 235024:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10015.241172] LustreError: 235024:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10017.797773] LustreError: 235010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857163 with bad export cookie 10981548067847066402 [10017.805221] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10017.817664] LustreError: 235010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10018.242985] Lustre: server umount lustre-MDT0001 complete [10032.166631] Lustre: server umount lustre-OST0000 complete [10036.193443] Lustre: 228340:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857165/real 1787857165] req@ffff8bc68a43b0c0 x1874703803339264/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857181 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10036.217822] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10037.422243] Lustre: server umount lustre-OST0001 complete [10044.936827] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [10056.812565] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [10098.565930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10098.836861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10098.869766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10098.955938] Lustre: lustre-MDT0000: new disk, initializing [10099.030794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10099.068217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10104.716749] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10115.754833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10115.915539] Lustre: 241283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10115.973458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10115.976980] Lustre: Skipped 1 previous similar message [10116.066617] Lustre: lustre-MDT0001: new disk, initializing [10116.303975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10116.328033] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10120.826151] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10129.899933] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10135.337356] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10142.873805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10143.151910] Lustre: lustre-OST0000: new disk, initializing [10143.155836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10143.174525] Lustre: 243263:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10145.159976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10145.169941] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10145.314905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10150.571372] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10160.870610] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10166.340423] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10166.545955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10171.880432] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10172.154223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10175.972644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10175.973516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10175.979649] LustreError: Skipped 1 previous similar message [10175.986449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10175.998883] Lustre: Skipped 1 previous similar message [10176.029873] Lustre: Skipped 1 previous similar message [10188.256090] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10188.496982] Lustre: server umount lustre-OST0000 complete [10192.868768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10192.876678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10198.789638] Lustre: server umount lustre-MDT0000 complete [10201.569892] LustreError: 241294:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10201.590358] LustreError: 241294:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10203.116989] LustreError: 241274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857348 with bad export cookie 10981548067847068537 [10203.119745] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10203.137035] LustreError: 241274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10203.670856] Lustre: server umount lustre-MDT0001 complete [10213.438322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10213.936186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10213.946078] Lustre: Skipped 2 previous similar messages [10218.997647] LustreError: 245483:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10219.029633] LustreError: 245483:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [10219.716661] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10229.651910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10234.941809] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10243.513939] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10249.481261] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10256.316216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10262.862711] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10271.146407] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10418.665091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10418.668133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10418.668760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10418.668766] Lustre: Skipped 7 previous similar messages [10418.703715] Lustre: Skipped 2 previous similar messages [10424.337745] Lustre: server umount lustre-OST0000 complete [10434.684094] Lustre: server umount lustre-MDT0000 complete [10438.983784] LustreError: 246220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857584 with bad export cookie 10981548067847069412 [10438.991507] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10439.000437] LustreError: 246220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10439.582748] Lustre: server umount lustre-MDT0001 complete [10539.497958] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [10542.146871] Key type lgssc unregistered [10542.603441] LNet: 249091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10542.621964] LNetError: 249091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10542.646115] LNet: Removed LNI 192.168.204.109@tcp [10543.496538] Key type .llcrypt unregistered [10543.501433] Key type ._llcrypt unregistered [10571.938513] Key type ._llcrypt registered [10571.941367] Key type .llcrypt registered [10572.105301] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [10591.669674] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [10593.345368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10593.361353] alg: No test for adler32 (adler32-zlib) [10594.721197] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [10595.027464] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [10596.799490] Key type lgssc registered [10598.183474] Lustre: Echo OBD driver; http://www.lustre.org/ [10651.212879] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [10662.325750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10662.352475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10663.657530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10663.694950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10663.817315] Lustre: lustre-MDT0000: new disk, initializing [10663.924318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10663.952689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10668.614365] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10678.431756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10678.558581] Lustre: 253171:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10678.609184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10678.613389] Lustre: Skipped 1 previous similar message [10678.681524] Lustre: lustre-MDT0001: new disk, initializing [10678.809101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10678.842366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10678.850447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10682.530812] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10687.805584] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10696.153023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10696.449231] Lustre: lustre-OST0000: new disk, initializing [10696.452308] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10696.458190] Lustre: 254803:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10696.536871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10700.535518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10700.547170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10700.708834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10703.332369] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10717.146875] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10717.465455] Lustre: lustre-OST0001: new disk, initializing [10717.470534] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10717.476316] Lustre: 255672:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10717.645594] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10725.039330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10725.067851] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10725.236139] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10726.075479] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10738.996311] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10751.320612] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10770.915499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10770.927330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10770.943810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10771.433876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10771.472951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10771.482259] Lustre: Skipped 3 previous similar messages [10773.235296] Lustre: server umount lustre-MDT0000 complete [10778.725187] LustreError: 253162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787857924 with bad export cookie 16720183252467796657 [10778.733248] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10778.763201] LustreError: 253162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10779.411612] Lustre: server umount lustre-MDT0001 complete [10793.636842] Lustre: server umount lustre-OST0000 complete [10797.472241] Lustre: 250671:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787857927/real 1787857927] req@ffff8bc68c5d83c0 x1874704717664128/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787857943 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10797.504183] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10797.515690] Lustre: Skipped 2 previous similar messages [10797.843756] Lustre: server umount lustre-OST0001 complete [10815.150422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10827.324643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10837.436967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10848.908876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10870.566846] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [10884.588793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10884.619947] Lustre: lustre-MDT0000: reset Object Index mappings [10885.453945] LustreError: 260686:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10885.483283] LustreError: 260686:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [10885.596765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10890.726516] LustreError: 260687:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10891.473996] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10895.857606] LustreError: 260686:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10900.961043] LustreError: 260687:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10902.522660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10902.536499] Lustre: lustre-MDT0001: reset Object Index mappings [10903.185252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10909.109203] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10912.889368] Lustre: 261829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10921.142179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10921.523275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10927.662393] LustreError: 262185:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10928.079173] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10930.762788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10935.788667] LustreError: 262185:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10935.827414] LustreError: 262185:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 3 previous similar messages [10939.525914] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10945.024218] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10948.659566] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [10958.267601] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10961.870171] Lustre: 263703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10967.662394] Lustre: server umount lustre-MDT0000 complete [10969.584709] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10969.597089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10969.622315] LustreError: 260687:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10972.200508] LustreError: 260668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858117 with bad export cookie 16720183252467803447 [10972.216590] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10972.227830] LustreError: 260668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10972.807918] Lustre: server umount lustre-MDT0001 complete [10988.434252] Lustre: server umount lustre-OST0000 complete [10992.228871] Lustre: 250673:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787858121/real 1787858121] req@ffff8bc6925d30c0 x1874704717718144/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787858137 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10992.255651] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10992.268726] Lustre: Skipped 3 previous similar messages [10992.494370] Lustre: server umount lustre-OST0001 complete [11016.832508] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 15:15:59 (1787858159) [11039.789868] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [11051.068507] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [11102.483944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11102.741160] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11102.823334] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11102.932615] Lustre: lustre-MDT0000: new disk, initializing [11103.002156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11103.009184] Lustre: Skipped 1 previous similar message [11103.021945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11109.749770] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11122.903415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11123.061250] Lustre: 267958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11123.120455] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11123.127466] Lustre: Skipped 1 previous similar message [11123.224422] Lustre: lustre-MDT0001: new disk, initializing [11123.379810] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11123.395196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11128.706334] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11139.883521] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11146.259527] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11154.370163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11154.664484] Lustre: lustre-OST0000: new disk, initializing [11154.668881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11154.686468] Lustre: 269927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11156.721376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11156.737140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11156.892303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11163.378751] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11174.789741] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11181.824164] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11182.153659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11188.030894] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11188.323958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11192.808454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11192.823440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11192.843589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11197.930781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11197.937910] Lustre: Skipped 1 previous similar message [11203.055884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11203.061904] Lustre: Skipped 2 previous similar messages [11205.087281] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11205.310184] Lustre: server umount lustre-OST0000 complete [11210.209331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11210.225890] LustreError: Skipped 1 previous similar message [11210.237626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11210.266658] Lustre: Skipped 1 previous similar message [11213.282703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11213.293975] Lustre: Skipped 2 previous similar messages [11216.324121] Lustre: server umount lustre-MDT0000 complete [11218.400645] LustreError: 267969:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11218.427638] LustreError: 267969:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [11221.147913] LustreError: 267950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858366 with bad export cookie 16720183252467805540 [11221.162256] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11221.164696] LustreError: 267950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11221.709949] Lustre: server umount lustre-MDT0001 complete [11244.529345] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [11260.104960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11260.852047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11260.873226] Lustre: Skipped 2 previous similar messages [11268.339054] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11280.080922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11287.554592] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11291.491596] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11299.633548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11308.137523] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11320.276484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11320.486933] Lustre: lustre-OST0001: new disk, initializing [11320.489533] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11320.495477] Lustre: 275179:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11322.154883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11322.168454] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11322.309366] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11328.533988] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11339.899398] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11344.387337] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11350.923694] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11350.936242] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11358.188826] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11358.190974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11358.219934] Lustre: Skipped 2 previous similar messages [11364.378216] Lustre: server umount lustre-MDT0000 complete [11368.424232] LustreError: 274372:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11368.439988] LustreError: 274372:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [11369.075541] LustreError: 272849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858514 with bad export cookie 16720183252467806415 [11369.088351] LustreError: 272849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11369.087759] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11369.530328] Lustre: server umount lustre-MDT0001 complete [11383.764741] Lustre: server umount lustre-OST0000 complete [11397.723993] Lustre: server umount lustre-OST0001 complete [11417.985541] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [11429.445582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11429.929210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11429.935535] Lustre: Skipped 3 previous similar messages [11434.746568] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11444.808908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11445.218172] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11445.221613] Lustre: Skipped 7 previous similar messages [11451.252355] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11454.828713] Lustre: 279375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11462.035946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11469.277351] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11474.692577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11479.132214] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11484.687932] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11486.193580] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11494.231296] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11498.701215] Lustre: 281252:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11504.163947] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11504.294401] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11514.576943] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 15:24:18 (1787858658) [11530.742534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11530.746935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11530.774754] Lustre: Skipped 4 previous similar messages [11530.792145] Lustre: Skipped 3 previous similar messages [11531.949934] Lustre: server umount lustre-MDT0000 complete [11535.863828] LustreError: 278230:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11535.884641] LustreError: 278230:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 9 previous similar messages [11536.015345] LustreError: 278215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858681 with bad export cookie 16720183252467812757 [11536.017685] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11536.026494] LustreError: 278215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11542.505967] Lustre: server umount lustre-MDT0001 complete [11552.916281] Lustre: server umount lustre-OST0000 complete [11557.062673] Lustre: server umount lustre-OST0001 complete [11564.832290] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [11579.336560] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [11638.088740] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [11650.452810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11650.799441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11650.839961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11650.931655] Lustre: lustre-MDT0000: new disk, initializing [11651.060632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11656.611526] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11668.358792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11668.489721] Lustre: 285495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11668.758579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11673.622375] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11686.104774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11686.439686] Lustre: 287125:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11686.552773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11686.563305] Lustre: Skipped 5 previous similar messages [11688.589767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11693.501091] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11704.518780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11704.743673] Lustre: 288000:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11706.889414] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11706.896237] Lustre: Skipped 1 previous similar message [11707.021357] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11711.899799] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11723.428173] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11727.635129] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11727.643090] Lustre: Skipped 1 previous similar message [11744.346306] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 15:28:08 (1787858888) [11747.819228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11747.834770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11747.854543] Lustre: Skipped 5 previous similar messages [11747.879578] Lustre: Skipped 7 previous similar messages [11748.945049] Lustre: server umount lustre-MDT0000 complete [11752.998778] LustreError: 287999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787858898 with bad export cookie 16720183252467820632 [11753.000690] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11753.012934] LustreError: 287999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11753.564625] Lustre: server umount lustre-MDT0001 complete [11768.437314] Lustre: server umount lustre-OST0000 complete [11781.988725] Lustre: server umount lustre-OST0001 complete [11791.499611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11792.040124] LustreError: 290245:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11792.067953] LustreError: 290245:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 8 previous similar messages [11797.207181] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11807.096739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11813.184976] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11826.261968] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11833.691715] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11842.248422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11851.093432] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11861.709204] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11882.672424] loop1: detected capacity change from 0 to 204800000 [11882.983436] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11882.991511] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11883.006767] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11889.578904] loop1: detected capacity change from 0 to 204800000 [11896.631180] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11896.735044] LustreError: 290235:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs22460: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11896.751945] LustreError: 290235:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs22460 [11896.756452] LustreError: 290235:0:(mgs_handler.c:517:mgs_target_reg()) fs22460-MDT0001: Error checking target index: rc = -13 [11896.768346] LustreError: fs22460-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11896.775617] LustreError: 293969:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11896.782800] LustreError: 293969:0:(tgt_mount.c:2129:server_put_super()) no obd fs22460-MDT0001 [11896.791552] LustreError: 293969:0:(tgt_mount.c:116:server_deregister_mount()) fs22460-MDT0001 not registered [11896.828949] Lustre: server umount fs22460-MDT0001 complete [11896.832904] LustreError: 293969:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11905.465277] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11905.677090] Lustre: Setting parameter fs22460-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs22460-MDT0001 [11905.756722] Lustre: srv-fs22460-MDT0001: No data found on store. Initialize space. [11905.765241] Lustre: Skipped 4 previous similar messages [11905.909493] Lustre: fs22460-MDT0001: new disk, initializing [11905.919894] Lustre: Skipped 3 previous similar messages [11906.087552] Lustre: srv-fs22460-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11906.098480] Lustre: 294435:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22460-MDT0001-mdtlov: can't create new llog: rc = -115 [11906.109245] LustreError: 294435:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22460-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11910.891140] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11918.061980] Lustre: server umount fs22460-MDT0001 complete [11931.047126] loop1: detected capacity change from 0 to 204800000 [11933.172695] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11936.921955] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11937.393876] Lustre: 296004:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22460-MDT0001-mdtlov: can't create new llog: rc = -115 [11937.408855] LustreError: 296004:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22460-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [11938.284597] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11942.477288] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11947.490938] Lustre: server umount fs22460-MDT0001 complete [11961.312314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11974.115483] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11974.539579] Lustre: server umount lustre-MDT0000 complete [11984.820602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11985.032536] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11990.878094] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [11991.027509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [12000.276785] loop1: detected capacity change from 0 to 204800000 [12006.641420] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12007.055703] Lustre: 298452:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22460-MDT0001-mdtlov: can't create new llog: rc = -115 [12007.067848] LustreError: 298452:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22460-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12012.334879] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12018.294203] Lustre: server umount fs22460-MDT0001 complete [12046.820354] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12046.825340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12046.832074] Lustre: Skipped 3 previous similar messages [12046.835431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12046.839298] Lustre: Skipped 14 previous similar messages [12049.605845] Lustre: server umount lustre-OST0000 complete [12057.577177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12068.319139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12068.484309] Lustre: server umount lustre-MDT0000 complete [12072.059774] LustreError: 290227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859217 with bad export cookie 16720183252467823313 [12072.064327] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12072.068542] LustreError: 290227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12072.390412] Lustre: server umount lustre-MDT0001 complete [12082.760268] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [12085.895745] Key type lgssc unregistered [12086.238089] LNet: 300801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12086.246778] LNetError: 300801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12086.262853] LNet: Removed LNI 192.168.204.109@tcp [12087.302170] Key type .llcrypt unregistered [12087.304853] Key type ._llcrypt unregistered [12109.146785] Key type ._llcrypt registered [12109.148985] Key type .llcrypt registered [12109.304372] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [12110.512978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12110.602529] alg: No test for adler32 (adler32-zlib) [12111.857193] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [12112.213693] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [12113.919502] Key type lgssc registered [12115.350771] Lustre: Echo OBD driver; http://www.lustre.org/ [12123.470352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12123.505938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12125.162300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12130.753283] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12135.395871] LustreError: 302070:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12135.411587] LustreError: 302070:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [12140.542113] LustreError: 302071:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12142.345369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12143.000923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12148.562688] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12157.932884] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12164.293248] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12172.772032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12173.215322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12181.277316] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12187.559907] LustreError: 303904:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12190.033849] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12192.758902] LustreError: 303905:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12199.930666] LustreError: 303902:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12199.956289] LustreError: 303902:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [12210.159799] LustreError: 303905:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12210.191353] LustreError: 303905:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 5 previous similar messages [12213.913407] loop1: detected capacity change from 0 to 204800000 [12214.154972] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12214.164190] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12214.172565] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12220.049292] loop1: detected capacity change from 0 to 204800000 [12226.345361] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12226.461914] Lustre: fs8786-OST0000: new disk, initializing [12226.465440] Lustre: srv-fs8786-OST0000: No data found on store. Initialize space. [12226.470772] Lustre: 305808:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12226.558403] Lustre: fs8786-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12226.565437] LustreError: 304522:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12226.584599] LustreError: 304522:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 9 previous similar messages [12233.302533] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12241.513550] Lustre: server umount fs8786-OST0000 complete [12252.636846] loop1: detected capacity change from 0 to 204800000 [12258.270732] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12258.551345] Lustre: fs8786-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12261.359330] LustreError: 303904:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12261.397098] LustreError: 303904:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 19 previous similar messages [12265.157727] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12266.468966] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12271.605258] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12275.840062] loop2: detected capacity change from 0 to 204800000 [12276.097736] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12276.127743] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12276.143876] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12282.440375] loop2: detected capacity change from 0 to 204800000 [12288.769172] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12288.843170] LustreError: fs8786-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12288.853743] LustreError: fs8786-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12288.864270] LustreError: 308848:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [12288.873263] LustreError: 308848:0:(tgt_mount.c:2129:server_put_super()) no obd fs8786-OST0001 [12288.884662] LustreError: 308848:0:(tgt_mount.c:116:server_deregister_mount()) fs8786-OST0001 not registered [12288.918237] Lustre: server umount fs8786-OST0001 complete [12288.927263] LustreError: 308848:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12300.730910] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12300.859304] Lustre: fs8786-OST0001: new disk, initializing [12300.866594] Lustre: srv-fs8786-OST0001: No data found on store. Initialize space. [12300.872745] Lustre: 309400:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12300.942431] Lustre: fs8786-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12307.513731] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12316.219211] Lustre: server umount fs8786-OST0000 complete [12326.354840] Lustre: server umount fs8786-OST0001 complete [12327.907961] LustreError: 303902:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12327.937147] LustreError: 303902:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 38 previous similar messages [12339.842292] loop1: detected capacity change from 0 to 204800000 [12347.402502] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12347.562097] Lustre: fs8786-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12356.425260] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12363.642818] loop2: detected capacity change from 0 to 204800000 [12369.506434] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12377.174745] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12385.948460] Lustre: server umount fs8786-OST0000 complete [12393.793809] Lustre: server umount fs8786-OST0001 complete [12407.776280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12407.781693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12407.794491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12409.827166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12409.829541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12409.829991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12409.839329] Lustre: Skipped 1 previous similar message [12412.613787] Lustre: server umount lustre-OST0000 complete [12419.553528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12419.562599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12419.573284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12423.409205] Lustre: server umount lustre-MDT0000 complete [12427.386082] LustreError: 302050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859572 with bad export cookie 4350473239907334579 [12427.394428] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12427.411141] LustreError: 302050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12427.790396] Lustre: server umount lustre-MDT0001 complete [12439.293605] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [12442.438634] Key type lgssc unregistered [12442.731822] LNet: 315328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12442.736596] LNetError: 315328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12442.750812] LNet: Removed LNI 192.168.204.109@tcp [12443.739266] Key type .llcrypt unregistered [12443.742766] Key type ._llcrypt unregistered [12484.757487] Key type ._llcrypt registered [12484.762562] Key type .llcrypt registered [12484.927964] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [12500.301290] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [12501.604881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12501.621274] alg: No test for adler32 (adler32-zlib) [12502.840624] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [12503.166637] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [12504.815181] Key type lgssc registered [12506.087872] Lustre: Echo OBD driver; http://www.lustre.org/ [12551.579379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12551.627982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12552.938473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12552.962926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12553.091804] Lustre: lustre-MDT0000: new disk, initializing [12553.320268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12553.374687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12557.725655] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12569.686053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12569.878263] Lustre: 319189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12569.979406] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12569.984415] Lustre: Skipped 1 previous similar message [12570.097392] Lustre: lustre-MDT0001: new disk, initializing [12570.186554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12570.214774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12570.227179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12575.236390] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12586.389254] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12592.702475] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12600.882443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12601.175978] Lustre: lustre-OST0000: new disk, initializing [12601.180763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12601.192033] Lustre: 321160:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12601.292439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12607.920654] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12608.101479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12608.117519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12608.310238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12617.977979] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12620.118441] Lustre: Mounted lustre-client - version 2.17.57_86_g6a41f06 [12621.348919] Lustre: Unmounted lustre-client [12634.082123] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12634.085505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12634.087587] LustreError: Skipped 1 previous similar message [12634.088477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12634.121669] Lustre: Skipped 1 previous similar message [12635.621659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12635.628295] Lustre: Skipped 2 previous similar messages [12638.205540] Lustre: server umount lustre-OST0000 complete [12645.875535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12645.883989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12645.897569] Lustre: Skipped 1 previous similar message [12645.910948] Lustre: Skipped 1 previous similar message [12649.329353] Lustre: server umount lustre-MDT0000 complete [12653.800803] LustreError: 319184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787859799 with bad export cookie 16149974856801921722 [12653.805167] LustreError: 319184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12653.805308] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12654.324975] Lustre: server umount lustre-MDT0001 complete [12667.165470] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [12669.929324] Key type lgssc unregistered [12672.287747] LNet: 323325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12672.298637] LNetError: 323325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12752.167683] LNet: Removed LNI 192.168.204.109@tcp [12753.589443] Key type .llcrypt unregistered [12753.597380] Key type ._llcrypt unregistered [12774.957553] Key type ._llcrypt registered [12774.960903] Key type .llcrypt registered [12775.133657] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [12794.031652] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [12795.388594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12795.406337] alg: No test for adler32 (adler32-zlib) [12796.392429] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [12796.544326] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [12798.183203] Key type lgssc registered [12799.333353] Lustre: Echo OBD driver; http://www.lustre.org/ [12831.381842] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 15:46:14 (1787859974) [12840.286631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12840.353902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12841.870166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12841.933414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12842.139933] Lustre: lustre-MDT0000: new disk, initializing [12842.293127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12842.324761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12848.764448] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12862.667608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12862.869496] Lustre: 326547:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12862.946369] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12862.961738] Lustre: Skipped 1 previous similar message [12863.111435] Lustre: lustre-MDT0001: new disk, initializing [12863.259488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12863.300223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12863.310722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12868.478405] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12879.591724] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12886.251721] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12893.875359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12894.257894] Lustre: lustre-OST0000: new disk, initializing [12894.263594] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12894.271415] Lustre: 328520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12894.397863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12901.483257] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [12903.526840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12903.535046] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12903.672108] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12911.152666] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12929.508941] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12929.518231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12929.525504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12929.536042] Lustre: Skipped 1 previous similar message [12929.551135] Lustre: Skipped 1 previous similar message [12934.546854] Lustre: server umount lustre-OST0000 complete [12939.745083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12939.785599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12939.797191] Lustre: Skipped 1 previous similar message [12944.864597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12944.876876] Lustre: Skipped 1 previous similar message [12949.986885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12949.998450] Lustre: Skipped 1 previous similar message [12953.055776] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12953.323582] Lustre: server umount lustre-MDT0000 complete [12958.036676] LustreError: 326542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860103 with bad export cookie 9959925101138926134 [12958.049307] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12958.066992] LustreError: 326542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12958.968290] Lustre: server umount lustre-MDT0001 complete [12973.114173] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [12976.574980] Key type lgssc unregistered [12976.976802] LNet: 330433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12976.990049] LNetError: 330433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12977.014771] LNet: Removed LNI 192.168.204.109@tcp [12978.121452] Key type .llcrypt unregistered [12978.127721] Key type ._llcrypt unregistered [13001.756919] Key type ._llcrypt registered [13001.760351] Key type .llcrypt registered [13001.898732] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [13002.870184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13002.877872] alg: No test for adler32 (adler32-zlib) [13004.032807] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [13004.281853] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [13005.944127] Key type lgssc registered [13007.216462] Lustre: Echo OBD driver; http://www.lustre.org/ [13016.675569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13016.705349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13018.315100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13024.505683] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13027.815985] LustreError: 331710:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13027.858712] LustreError: 331710:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13032.930702] LustreError: 331711:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13035.499846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13035.947150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13042.142971] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13051.227920] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13057.999329] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13065.884529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13066.554319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13074.829384] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13084.259871] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13100.521610] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13102.402689] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13106.738352] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.9@tcp (stopping) [13107.682120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13107.687240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13107.696539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13108.705635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13108.710727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13111.816413] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.9@tcp (stopping) [13111.825283] Lustre: Skipped 2 previous similar messages [13113.011423] Lustre: server umount lustre-MDT0000 complete [13113.833829] LustreError: 331710:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13113.853170] LustreError: 331710:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 2 previous similar messages [13116.935952] LustreError: 331707:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13117.097363] LustreError: 331690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860262 with bad export cookie 7182280104414179945 [13117.102657] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13117.118773] LustreError: 331690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13118.945778] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13118.964461] Lustre: Skipped 1 previous similar message [13118.974183] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13122.056575] LustreError: 332638:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13122.074310] LustreError: 332638:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [13123.543884] Lustre: server umount lustre-MDT0001 complete [13127.603431] Lustre: server umount lustre-OST0000 complete [13136.815703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13141.775874] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13145.073122] Lustre: 335906:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.109@tcp [13147.455516] Lustre: server umount MGS complete [13156.981829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13157.762921] LustreError: 336478:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13157.781961] LustreError: 336478:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 1 previous similar message [13157.836806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13163.021704] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13172.838293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13173.414089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13178.770347] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13188.075814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13188.426490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13196.416930] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13205.232676] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13207.328908] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13214.375565] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13216.400734] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13223.401771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13223.408278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13223.425119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13223.432554] Lustre: Skipped 2 previous similar messages [13228.925104] Lustre: server umount lustre-OST0000 complete [13233.632777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13233.662183] Lustre: Skipped 2 previous similar messages [13239.536693] Lustre: server umount lustre-MDT0000 complete [13243.873301] LustreError: 336474:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13243.895902] LustreError: 336474:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 6 previous similar messages [13244.484926] LustreError: 336457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860390 with bad export cookie 7182280104414180778 [13244.499646] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13244.504333] LustreError: 336457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13245.327708] Lustre: server umount lustre-MDT0001 complete [13260.298486] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [13263.573790] Key type lgssc unregistered [13263.814837] LNet: 339918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13263.825194] LNetError: 339918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13263.842809] LNet: Removed LNI 192.168.204.109@tcp [13264.752891] Key type .llcrypt unregistered [13264.759809] Key type ._llcrypt unregistered [13287.890930] Key type ._llcrypt registered [13287.892366] Key type .llcrypt registered [13288.025657] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [13306.862344] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [13309.126697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13309.159252] alg: No test for adler32 (adler32-zlib) [13310.321450] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [13310.621763] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [13312.337274] Key type lgssc registered [13313.571663] Lustre: Echo OBD driver; http://www.lustre.org/ [13356.731764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13356.762601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13358.018481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13358.076528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13358.215490] Lustre: lustre-MDT0000: new disk, initializing [13358.366845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13358.420050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13363.682588] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13375.750635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13375.889506] Lustre: 343084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13375.962301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13375.965487] Lustre: Skipped 1 previous similar message [13376.027473] Lustre: lustre-MDT0001: new disk, initializing [13376.096973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13376.146287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13376.170237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13381.082742] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13390.820089] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13397.475952] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13404.284172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13404.730332] Lustre: lustre-OST0000: new disk, initializing [13404.735567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13404.743142] Lustre: 345055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13404.860489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13411.477907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13411.492720] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13411.498339] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13411.688831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13422.829596] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13432.039925] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13432.665387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13438.031568] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13438.176705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13442.532851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13442.540581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13442.541843] LustreError: Skipped 1 previous similar message [13442.550847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13442.552715] Lustre: Skipped 1 previous similar message [13442.564830] Lustre: Skipped 1 previous similar message [13446.604869] Lustre: server umount lustre-OST0000 complete [13452.770149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13452.776875] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13452.791755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13452.824592] Lustre: Skipped 1 previous similar message [13457.226792] Lustre: server umount lustre-MDT0000 complete [13461.924130] LustreError: 343078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860607 with bad export cookie 541149884906963124 [13461.936905] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13461.946911] LustreError: 343078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13462.429101] Lustre: server umount lustre-MDT0001 complete [13473.188387] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 15:56:57 (1787860617) [13491.998937] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [13507.520416] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [13550.852752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13551.159212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13551.192970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13551.321675] Lustre: lustre-MDT0000: new disk, initializing [13551.461825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13551.497241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13557.032333] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13569.812845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13569.966392] Lustre: 349841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13570.065918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13570.072439] Lustre: Skipped 1 previous similar message [13570.175738] Lustre: lustre-MDT0001: new disk, initializing [13570.287503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13570.332923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13570.345791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13575.306156] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13585.408618] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13592.530197] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13600.534166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13600.864710] Lustre: lustre-OST0000: new disk, initializing [13600.868355] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13600.878086] Lustre: 351811:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13600.968712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13608.553310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13608.577234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13608.682993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13608.687537] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [13619.400475] Lustre: DEBUG MARKER: oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13665.253982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13665.268137] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13665.271237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13665.271259] Lustre: Skipped 1 previous similar message [13665.307301] Lustre: Skipped 1 previous similar message [13668.352766] Lustre: server umount lustre-OST0000 complete [13672.929308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13672.942557] LustreError: Skipped 1 previous similar message [13672.945372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13672.973379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13672.986320] Lustre: Skipped 1 previous similar message [13678.861223] Lustre: server umount lustre-MDT0000 complete [13680.613590] LustreError: 349848:0:(ldlm_lib.c:1200:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13680.655335] LustreError: 349848:0:(ldlm_lib.c:1200:target_handle_connect()) Skipped 4 previous similar messages [13683.498208] LustreError: 349833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787860829 with bad export cookie 541149884906964006 [13683.501705] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13683.506414] LustreError: 349833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13684.010831] Lustre: server umount lustre-MDT0001 complete [13697.778484] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [13700.751436] Key type lgssc unregistered [13701.079751] LNet: 353990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13701.091354] LNetError: 353990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13701.124609] LNet: Removed LNI 192.168.204.109@tcp [13702.039211] Key type .llcrypt unregistered [13702.042124] Key type ._llcrypt unregistered [13815.529829] Key type ._llcrypt registered [13815.533395] Key type .llcrypt registered [13815.650572] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [13815.981561] Key type .llcrypt unregistered [13815.986548] Key type ._llcrypt unregistered [13899.684995] Key type ._llcrypt registered [13899.691251] Key type .llcrypt registered [13899.794401] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [13900.049494] Key type .llcrypt unregistered [13900.053506] Key type ._llcrypt unregistered [13930.585176] Key type ._llcrypt registered [13930.586849] Key type .llcrypt registered [13930.765438] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [13931.274714] Key type .llcrypt unregistered [13931.276878] Key type ._llcrypt unregistered [13967.548922] Key type ._llcrypt registered [13967.551700] Key type .llcrypt registered [13967.696087] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [13968.241949] Key type .llcrypt unregistered [13968.248120] Key type ._llcrypt unregistered [14047.891318] Key type ._llcrypt registered [14047.894534] Key type .llcrypt registered [14048.027909] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [14048.444067] Key type .llcrypt unregistered [14048.446383] Key type ._llcrypt unregistered [14082.227964] Key type ._llcrypt registered [14082.238417] Key type .llcrypt registered [14082.429887] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [14082.874330] Key type .llcrypt unregistered [14082.875843] Key type ._llcrypt unregistered [14114.675720] Key type ._llcrypt registered [14114.679317] Key type .llcrypt registered [14114.816400] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [14115.137544] Key type .llcrypt unregistered [14115.145407] Key type ._llcrypt unregistered [14147.534790] Key type ._llcrypt registered [14147.537183] Key type .llcrypt registered [14147.648170] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [14148.003448] Key type .llcrypt unregistered [14148.006193] Key type ._llcrypt unregistered [14180.928808] Key type ._llcrypt registered [14180.931327] Key type .llcrypt registered [14181.025825] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [14181.307950] Key type .llcrypt unregistered [14181.312270] Key type ._llcrypt unregistered [14214.312975] Key type ._llcrypt registered [14214.319404] Key type .llcrypt registered [14214.443493] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [14214.774327] Key type .llcrypt unregistered [14214.775965] Key type ._llcrypt unregistered [14284.378312] Key type ._llcrypt registered [14284.393165] Key type .llcrypt registered [14284.500943] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [14284.896298] Key type .llcrypt unregistered [14284.897586] Key type ._llcrypt unregistered [14321.965834] Key type ._llcrypt registered [14321.968538] Key type .llcrypt registered [14322.111648] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [14322.409432] Key type .llcrypt unregistered [14322.412821] Key type ._llcrypt unregistered [14355.429737] Key type ._llcrypt registered [14355.434744] Key type .llcrypt registered [14355.583844] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [14355.929404] Key type .llcrypt unregistered [14355.933125] Key type ._llcrypt unregistered [14424.056272] Key type ._llcrypt registered [14424.062967] Key type .llcrypt registered [14424.202150] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [14443.033374] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [14443.950145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14444.018035] alg: No test for adler32 (adler32-zlib) [14445.284746] Lustre: Lustre: Build Version: 2.17.57_86_g6a41f06 [14445.539216] LNet: Added LNI 192.168.204.109@tcp [8/256/0/180] [14447.239185] Key type lgssc registered [14448.549961] Lustre: Echo OBD driver; http://www.lustre.org/ [14490.027101] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14490.081897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14491.375925] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14491.459162] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14491.634722] Lustre: tst250fs-MDT0000: new disk, initializing [14491.745309] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14491.768392] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14496.886080] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [14508.459072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14508.706262] Lustre: 374432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14508.791400] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14508.800199] Lustre: Skipped 1 previous similar message [14509.047174] Lustre: tst250fs-MDT0001: new disk, initializing [14509.163146] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14509.227819] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14509.262953] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14515.249832] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [14521.378533] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14528.790778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14529.127694] Lustre: tst250fs-OST0000: new disk, initializing [14529.134711] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14529.140591] Lustre: 376063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14529.258697] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14536.532894] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [14537.874421] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14537.881187] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14538.076728] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14547.896977] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14548.160061] Lustre: tst250fs-OST0001: new disk, initializing [14548.167513] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14548.180467] Lustre: 376937:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14548.341617] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14555.126892] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_default_debug -1 all [14557.329731] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14557.339699] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14557.411225] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14560.758162] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14563.763974] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14563.773155] Lustre: Skipped 2 previous similar messages [14583.266213] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14583.282961] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14583.304796] Lustre: Skipped 2 previous similar messages [14583.317096] Lustre: Skipped 1 previous similar message [14588.393603] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14588.408079] Lustre: Skipped 3 previous similar messages [14589.133908] Lustre: server umount tst250fs-MDT0000 complete [14593.353307] LustreError: 374425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787861738 with bad export cookie 14764823709768153657 [14593.362035] LustreError: MGC192.168.204.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14593.363960] LustreError: 374425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14593.505111] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14593.518040] Lustre: tst250fs-MDT0001: Not available for connect from 0@lo (stopping) [14593.530882] Lustre: Skipped 2 previous similar messages [14593.540975] Lustre: Skipped 1 previous similar message [14593.739171] Lustre: server umount tst250fs-MDT0001 complete [14600.175887] Lustre: server umount tst250fs-OST0000 complete [14604.599570] Lustre: server umount tst250fs-OST0001 complete [14629.352891] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [14639.400353] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [14677.366776] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 16:17:01 (1787861821) [14679.529814] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14697.493130] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing set_hostid [14709.465207] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing load_modules_local [14745.586576] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14466 sec ========= 16:18:09 (1787861889) [14748.004390] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:18:11 (1787861891) === [14752.744911] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:18:16 (1787861896) === [14795.107500] Lustre: DEBUG MARKER: oleg409-server.virtnet: executing unload_modules_local [14797.650464] Key type lgssc unregistered [14797.891916] LNet: 386084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14797.902639] LNetError: 386084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14797.939323] LNet: Removed LNI 192.168.204.109@tcp [14798.839180] Key type .llcrypt unregistered [14798.842411] Key type ._llcrypt unregistered