[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 505879024 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002377] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.007316] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010142] LSM: Security Framework initializing [ 0.011061] Yama: becoming mindful. [ 0.012045] SELinux: Initializing. [ 0.013074] *** VALIDATE selinux *** [ 0.022263] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027030] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028173] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030171] *** VALIDATE tmpfs *** [ 0.031422] *** VALIDATE proc *** [ 0.033073] *** VALIDATE cgroup *** [ 0.034014] *** VALIDATE cgroup2 *** [ 0.036108] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037172] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038018] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039036] Spectre V2 : User space: Vulnerable [ 0.041007] Speculative Store Bypass: Vulnerable [ 0.044079] debug: unmapping init [mem 0xffffffffa1059000-0xffffffffa1060fff] [ 0.047199] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048786] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049028] ... version: 2 [ 0.050019] ... bit width: 48 [ 0.051015] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053018] ... max period: 00007fffffffffff [ 0.054012] ... fixed-purpose events: 3 [ 0.055015] ... event mask: 000000070000000f [ 0.056301] rcu: Hierarchical SRCU implementation. [ 0.058485] smp: Bringing up secondary CPUs ... [ 0.059611] x86: Booting SMP configuration: [ 0.060036] .... node #0, CPUs: #1 #2 #3 [ 0.063610] smp: Brought up 1 node, 4 CPUs [ 0.065016] smpboot: Max logical packages: 1 [ 0.066022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132481] node 0 deferred pages initialised in 63ms [ 0.136119] devtmpfs: initialized [ 0.137303] x86/mm: Memory block size: 128MB [ 0.140964] gcov: version magic: 0x41383552 [ 0.143248] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144069] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.145243] pinctrl core: initialized pinctrl subsystem [ 0.147228] [ 0.147897] ************************************************************* [ 0.150019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.153013] ** ** [ 0.156014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158015] ** ** [ 0.161015] ** This means that this kernel is built to expose internal ** [ 0.164015] ** IOMMU data structures, which may compromise security on ** [ 0.166014] ** your system. ** [ 0.169017] ** ** [ 0.172023] ** If you see this message and you are not debugging the ** [ 0.174012] ** kernel, report this immediately to your vendor! ** [ 0.177020] ** ** [ 0.179012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182014] ************************************************************* [ 0.184689] NET: Registered protocol family 16 [ 0.186537] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.190065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196459] cpuidle: using governor menu [ 0.197604] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.201026] PCI: Using configuration type 1 for base access [ 0.203154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217136] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220044] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223092] cryptd: max_cpu_qlen set to 1000 [ 0.227018] ACPI: Added _OSI(Module Device) [ 0.228025] ACPI: Added _OSI(Processor Device) [ 0.229016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.236152] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.242562] ACPI: Interpreter enabled [ 0.244077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.246013] ACPI: Using IOAPIC for interrupt routing [ 0.248214] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.252499] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.263732] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.267078] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.271036] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.277098] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.281059] acpiphp: Slot [2] registered [ 0.282134] acpiphp: Slot [5] registered [ 0.284130] acpiphp: Slot [6] registered [ 0.285135] acpiphp: Slot [7] registered [ 0.286108] acpiphp: Slot [8] registered [ 0.287133] acpiphp: Slot [9] registered [ 0.288131] acpiphp: Slot [10] registered [ 0.289151] acpiphp: Slot [3] registered [ 0.291113] acpiphp: Slot [4] registered [ 0.292000] acpiphp: Slot [11] registered [ 0.292000] acpiphp: Slot [12] registered [ 0.294105] acpiphp: Slot [13] registered [ 0.295111] acpiphp: Slot [14] registered [ 0.296097] acpiphp: Slot [15] registered [ 0.298112] acpiphp: Slot [16] registered [ 0.299094] acpiphp: Slot [17] registered [ 0.301188] acpiphp: Slot [18] registered [ 0.302182] acpiphp: Slot [19] registered [ 0.306185] acpiphp: Slot [20] registered [ 0.309197] acpiphp: Slot [21] registered [ 0.311147] acpiphp: Slot [22] registered [ 0.312147] acpiphp: Slot [23] registered [ 0.314184] acpiphp: Slot [24] registered [ 0.317063] acpiphp: Slot [25] registered [ 0.319181] acpiphp: Slot [26] registered [ 0.321136] acpiphp: Slot [27] registered [ 0.323125] acpiphp: Slot [28] registered [ 0.324096] acpiphp: Slot [29] registered [ 0.325094] acpiphp: Slot [30] registered [ 0.327144] acpiphp: Slot [31] registered [ 0.329069] PCI host bridge to bus 0000:00 [ 0.330019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.335027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.337028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.341036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.343027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345166] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.348011] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.351587] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.364745] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.371066] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.374108] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.378043] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.383055] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.387033] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.390930] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.395058] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.399200] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.407026] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.423028] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.429019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.435518] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.445028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.455025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.475106] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.490281] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.497030] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.504028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.527020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.536892] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.550018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.559023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.579029] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.596398] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.603023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.610022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.649024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.669370] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.678014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.684029] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.708017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.722680] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.735023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.744018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.771017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.783819] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.785368] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.788371] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.791454] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.793250] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.799040] iommu: Default domain type: Passthrough [ 0.802442] SCSI subsystem initialized [ 0.803140] ACPI: bus type USB registered [ 0.805333] usbcore: registered new interface driver usbfs [ 0.807088] usbcore: registered new interface driver hub [ 0.809096] usbcore: registered new device driver usb [ 0.811190] pps_core: LinuxPPS API ver. 1 registered [ 0.813013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.815083] PTP clock support registered [ 0.818037] EDAC MC: Ver: 3.0.0 [ 0.819504] PCI: Using ACPI for IRQ routing [ 0.820714] NetLabel: Initializing [ 0.822016] NetLabel: domain hash size = 128 [ 0.824014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.826084] NetLabel: unlabeled traffic allowed by default [ 0.829132] vgaarb: loaded [ 0.831230] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.833016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.839712] clocksource: Switched to clocksource kvm-clock [ 0.949806] VFS: Disk quotas dquot_6.6.0 [ 0.951405] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.954045] *** VALIDATE ramfs *** [ 0.955483] *** VALIDATE hugetlbfs *** [ 0.956753] pnp: PnP ACPI init [ 0.958924] pnp: PnP ACPI: found 6 devices [ 0.980662] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.984598] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.987342] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.990147] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.993121] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.995738] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.998681] NET: Registered protocol family 2 [ 1.001470] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.006719] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.011332] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.016445] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.020802] TCP: Hash tables configured (established 65536 bind 65536) [ 1.024199] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.027922] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.030845] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.033706] NET: Registered protocol family 1 [ 1.035976] RPC: Registered named UNIX socket transport module. [ 1.037936] RPC: Registered udp transport module. [ 1.039434] RPC: Registered tcp transport module. [ 1.040497] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.041918] NET: Registered protocol family 44 [ 1.043903] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.046507] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.049112] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.051632] PCI: CLS 0 bytes, default 64 [ 1.052991] Unpacking initramfs... [ 2.459972] debug: unmapping init [mem 0xffff8da8bcc54000-0xffff8da8bffbffff] [ 2.464553] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.467311] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.470734] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.964465] Initialise system trusted keyrings [ 2.966881] Key type blacklist registered [ 2.968939] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.978237] zbud: loaded [ 2.981242] *** VALIDATE nfs *** [ 2.983175] *** VALIDATE nfs4 *** [ 2.985034] pstore: using deflate compression [ 2.988298] Platform Keyring initialized [ 3.100778] NET: Registered protocol family 38 [ 3.103117] Key type asymmetric registered [ 3.105609] Asymmetric key parser 'x509' registered [ 3.108161] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.112206] io scheduler mq-deadline registered [ 3.114579] io scheduler kyber registered [ 3.116883] io scheduler bfq registered [ 3.119302] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.122438] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.125586] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.128786] ACPI: Power Button [PWRF] [ 3.135408] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.143792] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.164918] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.173163] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.192087] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.221203] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.249736] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.255531] Non-volatile memory driver v1.3 [ 3.257680] Linux agpgart interface v0.103 [ 3.294000] virtio_blk virtio1: [vda] 149808 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.297296] vda: detected capacity change from 0 to 76701696 [ 3.316744] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.319556] vdb: detected capacity change from 0 to 1073741824 [ 3.338864] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.342802] vdc: detected capacity change from 0 to 2621440000 [ 3.358695] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.361830] vdd: detected capacity change from 0 to 2621440000 [ 3.376154] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.379015] vde: detected capacity change from 0 to 4294967296 [ 3.396811] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.399753] vdf: detected capacity change from 0 to 4294967296 [ 3.407162] libphy: Fixed MDIO Bus: probed [ 3.414138] usbcore: registered new interface driver usbserial_generic [ 3.417099] usbserial: USB Serial support registered for generic [ 3.419821] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.425459] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.427557] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.430068] mousedev: PS/2 mouse device common for all mice [ 3.432858] rtc_cmos 00:05: RTC can wake from S4 [ 3.435878] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.437267] rtc_cmos 00:05: registered as rtc0 [ 3.443104] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.445145] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.447158] intel_pstate: CPU model not supported [ 3.452550] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.458360] hid: raw HID events driver (C) Jiri Kosina [ 3.460772] usbcore: registered new interface driver usbhid [ 3.462937] usbhid: USB HID core driver [ 3.464797] drop_monitor: Initializing network drop monitor service [ 3.467821] Initializing XFRM netlink socket [ 3.470050] NET: Registered protocol family 10 [ 3.473062] Segment Routing with IPv6 [ 3.474862] NET: Registered protocol family 17 [ 3.477048] mpls_gso: MPLS GSO support [ 3.482405] RAS: Correctable Errors collector initialized. [ 3.484665] AVX version of gcm_enc/dec engaged. [ 3.486340] AES CTR mode by8 optimization enabled [ 3.569316] sched_clock: Marking stable (3569274864, 0)->(4499809953, -930535089) [ 3.573976] registered taskstats version 1 [ 3.576341] Loading compiled-in X.509 certificates [ 3.578498] zswap: loaded using pool lzo/zbud [ 3.608712] Key type big_key registered [ 3.623096] Key type encrypted registered [ 3.625117] ima: No TPM chip found, activating TPM-bypass! [ 3.627554] ima: Allocated hash algorithm: sha1 [ 3.629667] ima: No architecture policies found [ 3.631149] evm: Initialising EVM extended attributes: [ 3.633260] evm: security.selinux [ 3.634678] evm: security.ima [ 3.635931] evm: security.capability [ 3.637744] evm: HMAC attrs: 0x1 [ 3.640314] rtc_cmos 00:05: setting system clock to 2026-09-06 01:27:06 UTC (1788658026) [ 3.646677] debug: unmapping init [mem 0xffffffffa2003000-0xffffffffa21fffff] [ 3.649935] debug: unmapping init [mem 0xffffffffa0d82000-0xffffffffa1058fff] [ 3.655089] Write protecting the kernel read-only data: 28672k [ 3.659113] debug: unmapping init [mem 0xffffffff9f403000-0xffffffff9f5fffff] [ 3.662219] debug: unmapping init [mem 0xffffffff9fd14000-0xffffffff9fdfffff] [ 3.700504] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.710673] systemd[1]: Detected virtualization kvm. [ 3.712861] systemd[1]: Detected architecture x86-64. [ 3.715544] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.744236] systemd[1]: No hostname configured. [ 3.746357] systemd[1]: Set hostname to . [ 3.749895] random: systemd: uninitialized urandom read (16 bytes read) [ 3.752774] systemd[1]: Initializing machine ID from random generator. [ 3.902801] random: systemd: uninitialized urandom read (16 bytes read) [ 3.906106] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.912508] random: systemd: uninitialized urandom read (16 bytes read) [ 3.915335] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.919672] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.625050] device-mapper: uevent: version 1.0.3 [ 4.627205] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.436182] virtio_net virtio0 ens2: renamed from eth0 [ 5.450349] random: fast init done [ 5.483909] scsi host0: ata_piix [ 5.530832] scsi host1: ata_piix [ 5.532227] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.534241] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.047267] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.144657] random: crng init done [ 10.146151] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.645219] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.794545] printk: systemd: 24 output lines suppressed due to ratelimiting [ 12.070572] SELinux: Disabled at runtime. [ 12.131951] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.140525] systemd[1]: Detected virtualization kvm. [ 12.142518] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.646384] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.651449] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.656981] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.661733] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.665188] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.671885] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.675489] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.722532] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.084947] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.379497] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.385976] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.510285] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.528913] EDAC sbridge: Ver: 1.1.2 [ 15.198349] Key type dns_resolver registered [ 15.516964] NFS: Registering the id_resolver key type [ 15.519151] Key type id_resolver registered [ 15.520720] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg442-server login: [ 53.796719] libcfs: loading out-of-tree module taints kernel. [ 53.862444] Key type ._llcrypt registered [ 53.864141] Key type .llcrypt registered [ 54.018548] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 73.069314] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 74.582533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 74.595389] alg: No test for adler32 (adler32-zlib) [ 76.066308] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 76.873236] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 78.623316] Key type lgssc registered [ 80.493599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 99.903543] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 140.309415] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 152.692662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 152.732819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 154.079752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 154.125770] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 154.231504] Lustre: lustre-MDT0000: new disk, initializing [ 154.302159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 154.343786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 158.956131] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 172.767679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.925153] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 172.956599] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 172.960631] Lustre: Skipped 1 previous similar message [ 173.133259] Lustre: lustre-MDT0001: new disk, initializing [ 173.252380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 173.305772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 173.322181] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 178.157690] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 183.725753] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 194.438145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 195.142510] Lustre: lustre-OST0000: new disk, initializing [ 195.146961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 195.153526] Lustre: 8446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 195.228317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.868108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 195.874850] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 195.959668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.573604] hrtimer: interrupt took 12593512 ns [ 202.115951] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.058639] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.193422] Lustre: lustre-OST0001: new disk, initializing [ 216.195918] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 216.201267] Lustre: 9519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.262827] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 222.857452] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.314529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 223.327442] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 223.362174] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 234.118754] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 240.283327] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 247.313975] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing check_logdir /tmp/testlogs/ [ 252.727519] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing yml_node [ 256.835103] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 259.376793] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 261.822404] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 263.322710] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 21:31:24 EDT 2026 [ 279.655564] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 281.341805] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 300.515045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 300.516885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 300.535696] Lustre: Skipped 1 previous similar message [ 300.548908] Lustre: Skipped 3 previous similar messages [ 303.917197] Lustre: server umount lustre-MDT0000 complete [ 309.121512] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658331 with bad export cookie 13641767262588344193 [ 309.125765] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 309.133138] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 310.757091] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.760151] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.760782] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.774252] LustreError: 6515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 310.808020] Lustre: Skipped 3 previous similar messages [ 314.785151] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 314.786453] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 314.829994] Lustre: Skipped 2 previous similar messages [ 315.583027] Lustre: server umount lustre-MDT0001 complete [ 320.009926] Lustre: server umount lustre-OST0000 complete [ 324.108703] Lustre: server umount lustre-OST0001 complete [ 329.912750] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 339.369735] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 372.724793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 372.945789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 372.969732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 373.038245] Lustre: lustre-MDT0000: new disk, initializing [ 373.099712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 373.128654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 377.510957] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 387.829402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 387.973465] Lustre: 15068:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 388.056255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 388.060823] Lustre: Skipped 1 previous similar message [ 388.201212] Lustre: lustre-MDT0001: new disk, initializing [ 388.316343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 388.360685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 388.379537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 392.182306] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 402.415332] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 408.816898] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 415.354877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 415.588994] Lustre: lustre-OST0000: new disk, initializing [ 415.593427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 415.599478] Lustre: 17041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 415.700929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.852459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 416.869053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 416.969067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 422.441246] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 432.084654] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 436.241522] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 436.463816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 440.411613] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 440.569409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 442.860756] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 442.861565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 442.877635] LustreError: Skipped 1 previous similar message [ 442.901118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 447.972306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 447.977778] Lustre: Skipped 2 previous similar messages [ 448.534151] Lustre: server umount lustre-OST0000 complete [ 453.099864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 453.113160] Lustre: Skipped 1 previous similar message [ 458.208219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 458.216185] Lustre: Skipped 3 previous similar messages [ 466.399146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 466.571472] Lustre: server umount lustre-MDT0000 complete [ 468.449250] LustreError: 15077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 468.469142] LustreError: 15077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 470.061881] LustreError: 15867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658492 with bad export cookie 13641767262588345726 [ 470.070073] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 470.073446] LustreError: 15867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 470.393719] Lustre: server umount lustre-MDT0001 complete [ 477.658326] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 21:34:57 (1788658497) [ 497.115149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 497.527451] LustreError: 19916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.615681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 501.255787] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 502.754936] LustreError: 19915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.372287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 514.666317] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 521.857311] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 527.044858] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 529.010477] Lustre: Failing over lustre-MDT0000 [ 529.211988] Lustre: server umount lustre-MDT0000 complete [ 530.402487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 530.417588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.429548] Lustre: Skipped 1 previous similar message [ 530.438285] LustreError: 19915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.469480] LustreError: 19915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 548.188050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.296903] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 552.717647] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 553.957838] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 553.973213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 554.010950] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 563.229651] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 569.315809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 569.324053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 569.327562] Lustre: Skipped 2 previous similar messages [ 569.333587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 569.347976] Lustre: Skipped 3 previous similar messages [ 573.566336] Lustre: server umount lustre-MDT0000 complete [ 574.433725] LustreError: 19912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 574.459147] LustreError: 19912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 576.663521] LustreError: 19897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658599 with bad export cookie 13641767262588347140 [ 576.665894] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 576.670815] LustreError: 19897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 576.920855] Lustre: server umount lustre-MDT0001 complete [ 583.685726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 584.091155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 584.097198] Lustre: Skipped 2 previous similar messages [ 587.989220] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 595.954546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 600.447682] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 607.180605] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 613.974384] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 616.476286] Lustre: Failing over lustre-MDT0000 [ 616.725246] Lustre: server umount lustre-MDT0000 complete [ 616.929310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 616.939223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.968194] LustreError: 23253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.983781] LustreError: 23253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 634.338193] Lustre: 3643:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788658641/real 1788658641] req@ffff8da937debb80 x1875543955002368/t0(0) o400->MGC192.168.204.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788658657 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 634.366383] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 635.235186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.576253] LustreError: 3639:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8da941bdd880 x1875543955005184/t0(0) o250->MGC192.168.204.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 644.875371] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 649.261386] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 650.212166] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 650.236782] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 650.243337] Lustre: Skipped 1 previous similar message [ 650.272593] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 658.731382] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 662.275846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 662.293442] Lustre: Skipped 1 previous similar message [ 662.304339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 662.309380] Lustre: Skipped 1 previous similar message [ 668.619278] Lustre: server umount lustre-MDT0000 complete [ 672.159562] LustreError: 23233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658695 with bad export cookie 13641767262588348078 [ 672.160817] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 672.171236] LustreError: 23233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 672.440854] Lustre: server umount lustre-MDT0001 complete [ 680.196317] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 21:38:21 (1788658701) [ 695.726784] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 705.453909] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 740.240799] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 749.210611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 749.367072] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 749.407963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 749.492926] Lustre: lustre-MDT0000: new disk, initializing [ 749.594468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 749.602661] Lustre: Skipped 2 previous similar messages [ 749.616749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 754.028658] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 764.482194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 764.579330] Lustre: 29628:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 764.843649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 768.883904] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 777.789387] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 783.284423] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 789.559779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 789.772559] Lustre: 31597:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 791.270084] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 791.308242] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 791.361705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 791.376524] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 794.742749] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 804.521624] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 812.610210] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 832.481063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 832.497885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 832.512890] Lustre: Skipped 1 previous similar message [ 832.527047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 832.530142] Lustre: Skipped 3 previous similar messages [ 845.279230] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 845.508174] Lustre: server umount lustre-OST0000 complete [ 851.940883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 851.948858] LustreError: Skipped 1 previous similar message [ 855.863927] Lustre: server umount lustre-MDT0000 complete [ 858.082590] LustreError: 29638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 858.101661] LustreError: 29638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 859.554264] LustreError: 29618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788658882 with bad export cookie 13641767262588348484 [ 859.557197] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 859.565545] LustreError: 29618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 859.971329] Lustre: server umount lustre-MDT0001 complete [ 870.734206] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 873.591842] Key type lgssc unregistered [ 873.859173] LNet: 33573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.863937] LNetError: 33573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.918747] LNet: Removed LNI 192.168.204.142@tcp [ 876.132230] Key type .llcrypt unregistered [ 876.136137] Key type ._llcrypt unregistered [ 904.149103] Key type ._llcrypt registered [ 904.151718] Key type .llcrypt registered [ 904.313622] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 905.752265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 905.786955] alg: No test for adler32 (adler32-zlib) [ 907.293367] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 907.651221] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 909.576270] Key type lgssc registered [ 910.624133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 917.929638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 917.989366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 919.433992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 923.669126] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 929.762382] LustreError: 34987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 929.801896] LustreError: 34987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 932.559450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 932.980226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 937.420638] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 945.630426] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 951.824813] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 958.502782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 958.854240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.832984] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 972.950452] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 977.328758] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 990.161738] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 21:43:30 (1788659010) [ 992.351553] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 994.647286] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1002.313441] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 21:43:42 (1788659022) [ 1013.091524] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 21:43:53 (1788659033) [ 1014.334241] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1024.023062] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1024.028778] Lustre: Skipped 2 previous similar messages [ 1026.043983] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 21:44:06 (1788659046) [ 1028.054830] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1028.061094] Lustre: Skipped 2 previous similar messages [ 1036.488508] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1036.499504] Lustre: Skipped 4 previous similar messages [ 1052.202642] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 21:44:32 (1788659072) [ 1055.155744] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1059.098636] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1068.894228] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 21:44:49 (1788659089) [ 1070.714634] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1070.719882] Lustre: Skipped 6 previous similar messages [ 1086.170266] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 21:45:06 (1788659106) [ 1103.328447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1103.335074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.346410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1105.387269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1105.393532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1105.418492] Lustre: Skipped 1 previous similar message [ 1108.600506] Lustre: server umount lustre-OST0000 complete [ 1115.628465] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.643949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1120.739847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1120.762726] Lustre: Skipped 2 previous similar messages [ 1125.863393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1125.869810] Lustre: Skipped 1 previous similar message [ 1127.391160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1127.795220] Lustre: server umount lustre-MDT0000 complete [ 1130.976847] LustreError: 41351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1131.017753] LustreError: 41351:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1132.341617] LustreError: 34969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788659155 with bad export cookie 2997336523922964975 [ 1132.356051] LustreError: 34969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1132.361338] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1132.838804] Lustre: server umount lustre-MDT0001 complete [ 1145.547769] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1148.208569] Key type lgssc unregistered [ 1148.489090] LNet: 42840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.498446] LNetError: 42840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1148.515268] LNet: Removed LNI 192.168.204.142@tcp [ 1149.265330] Key type .llcrypt unregistered [ 1149.268088] Key type ._llcrypt unregistered [ 1169.431835] Key type ._llcrypt registered [ 1169.434205] Key type .llcrypt registered [ 1169.570625] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 1187.366806] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1189.183620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1189.198546] alg: No test for adler32 (adler32-zlib) [ 1190.187283] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 1190.463883] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1192.159203] Key type lgssc registered [ 1193.593383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1239.602991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1239.644420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1240.855674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1240.891557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1240.986304] Lustre: lustre-MDT0000: new disk, initializing [ 1241.040578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1241.061077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1245.412759] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1257.364684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.546691] Lustre: 45993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1257.628447] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1257.638503] Lustre: Skipped 1 previous similar message [ 1257.756928] Lustre: lustre-MDT0001: new disk, initializing [ 1257.917487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1257.966747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1257.984330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1262.693075] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1275.185719] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1281.654118] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1287.959912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1288.141229] Lustre: lustre-OST0000: new disk, initializing [ 1288.143728] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1288.147285] Lustre: 47961:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1288.208099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1294.001446] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1298.002282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1298.014481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1298.178232] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1303.887471] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1308.851992] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1309.028995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1314.751677] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1315.036465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1318.891667] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1318.895720] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.901303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1318.940446] Lustre: Skipped 1 previous similar message [ 1323.386201] Lustre: server umount lustre-OST0000 complete [ 1329.126146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1329.129030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1329.143094] Lustre: Skipped 1 previous similar message [ 1329.168053] Lustre: Skipped 2 previous similar messages [ 1334.243118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1334.252652] Lustre: Skipped 1 previous similar message [ 1338.338165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1338.351473] Lustre: Skipped 1 previous similar message [ 1341.407117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1341.647543] Lustre: server umount lustre-MDT0000 complete [ 1345.757624] LustreError: 45985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788659368 with bad export cookie 15097993702651792082 [ 1345.768454] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1345.781111] LustreError: 45985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1346.140828] Lustre: server umount lustre-MDT0001 complete [ 1357.379478] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 21:49:37 (1788659377) [ 1365.170573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1365.695701] LustreError: 50425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1365.728632] LustreError: 50425:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1365.806996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.121884] LustreError: 50426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.498710] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1375.203374] LustreError: 50425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1380.324361] LustreError: 50426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1381.710095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1382.176244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1388.397808] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1398.265381] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1405.472530] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1413.445364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1414.000116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.145619] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1430.275869] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1439.535565] LustreError: 52957:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1440.719066] Lustre: Modifying parameter general.timeout=1 in log params [ 1441.785186] Lustre: Modifying parameter general.timeout=2 in log params [ 1443.810783] Lustre: Modifying parameter general.timeout=4 in log params [ 1443.817113] Lustre: Skipped 1 previous similar message [ 1448.394936] Lustre: Modifying parameter general.timeout=8 in log params [ 1448.403263] Lustre: Skipped 3 previous similar messages [ 1456.761195] Lustre: Modifying parameter general.timeout=14 in log params [ 1456.764877] Lustre: Skipped 5 previous similar messages [ 1473.521439] Lustre: Modifying parameter general.timeout=28 in log params [ 1473.538326] Lustre: Skipped 13 previous similar messages [ 1506.130188] Lustre: Modifying parameter general.timeout=60 in log params [ 1506.134735] Lustre: Skipped 31 previous similar messages [ 1570.873866] Lustre: Modifying parameter general.timeout=125 in log params [ 1570.878748] Lustre: Skipped 64 previous similar messages [ 1585.873531] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 21:53:26 (1788659606) [ 1588.748846] Lustre: 59915:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1590.824292] Lustre: 60015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1594.719874] Lustre: 60212:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1594.735734] Lustre: 60212:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1627.567386] Lustre: 61788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1627.577271] Lustre: 61788:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1627.601518] LustreError: 61788:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1632.531886] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 21:54:12 (1788659652) [ 1651.682343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1651.740424] Lustre: Skipped 2 previous similar messages [ 1651.748274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1651.758890] Lustre: Skipped 2 previous similar messages [ 1654.245269] Lustre: server umount lustre-MDT0000 complete [ 1656.802830] LustreError: 59985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1656.817876] LustreError: 59985:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1658.668562] LustreError: 55387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788659681 with bad export cookie 15097993702651792964 [ 1658.676778] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1658.687201] LustreError: 55387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1659.266509] Lustre: server umount lustre-MDT0001 complete [ 1674.194742] Lustre: server umount lustre-OST0000 complete [ 1704.553734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1705.087514] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1705.123239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1705.139745] Lustre: Skipped 27 previous similar messages [ 1705.481329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1710.100746] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1719.983583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1720.078045] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1720.099159] Lustre: 64325:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1724.927171] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1732.965581] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1738.942277] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1746.065274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1746.307520] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1746.607787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1746.614596] Lustre: Skipped 1 previous similar message [ 1754.135259] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1762.158659] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1763.755620] Lustre: 66893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1770.979385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1770.981797] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1770.982979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1770.995249] Lustre: Skipped 1 previous similar message [ 1771.016037] LustreError: Skipped 1 previous similar message [ 1776.842697] Lustre: server umount lustre-OST0000 complete [ 1781.222737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1781.236157] Lustre: Skipped 1 previous similar message [ 1786.739902] Lustre: server umount lustre-MDT0000 complete [ 1791.118351] LustreError: 64317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788659813 with bad export cookie 15097993702651796170 [ 1791.120355] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1791.125441] LustreError: 64317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1791.654304] Lustre: server umount lustre-MDT0001 complete [ 1803.841779] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1806.975651] Key type lgssc unregistered [ 1807.309062] LNet: 68103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.331619] LNetError: 68103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.424841] LNet: Removed LNI 192.168.204.142@tcp [ 1809.280933] Key type .llcrypt unregistered [ 1809.286094] Key type ._llcrypt unregistered [ 1834.570975] Key type ._llcrypt registered [ 1834.574537] Key type .llcrypt registered [ 1834.750751] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1835.754866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1835.880334] alg: No test for adler32 (adler32-zlib) [ 1836.925600] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 1837.201190] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1838.871186] Key type lgssc registered [ 1839.994702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1848.461897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1848.497115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1850.236738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.887222] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1859.561812] LustreError: 69514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1859.583208] LustreError: 69514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1864.407180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.687385] LustreError: 69515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1865.364879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1869.698796] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1877.383773] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1882.594522] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1889.158275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1889.848231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1897.087169] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1906.005540] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1919.292285] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1931.744574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1931.767387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1931.783712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1934.842784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1934.843578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1934.869817] Lustre: Skipped 1 previous similar message [ 1934.890327] Lustre: Skipped 2 previous similar messages [ 1937.455558] Lustre: server umount lustre-MDT0000 complete [ 1939.938056] LustreError: 69510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1939.965691] LustreError: 69510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1942.598266] LustreError: 69494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788659965 with bad export cookie 12277814431245983441 [ 1942.610637] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1942.615707] LustreError: 69494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1943.404222] Lustre: server umount lustre-MDT0001 complete [ 1958.664071] Lustre: server umount lustre-OST0000 complete [ 1991.887917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1992.351429] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1992.409671] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1992.721481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1997.978147] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2007.601248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2007.747609] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2007.766376] Lustre: 74389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2008.271605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2013.074390] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2020.995857] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2026.688843] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2034.147455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2034.398802] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2034.885303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2042.721421] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2051.630626] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2053.610978] Lustre: 76977:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2055.933495] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2063.340251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2063.361355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2063.402672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2068.962838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2068.980635] Lustre: Skipped 2 previous similar messages [ 2069.262924] Lustre: server umount lustre-OST0000 complete [ 2074.083413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.091926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2074.097724] Lustre: Skipped 2 previous similar messages [ 2074.108158] Lustre: Skipped 1 previous similar message [ 2079.732525] Lustre: server umount lustre-MDT0000 complete [ 2084.156762] LustreError: 75146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788660107 with bad export cookie 12277814431245984281 [ 2084.157703] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2084.163538] LustreError: 75146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2084.778921] Lustre: server umount lustre-MDT0001 complete [ 2098.038884] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2101.304677] Key type lgssc unregistered [ 2101.670222] LNet: 78185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.677786] LNetError: 78185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2101.697597] LNet: Removed LNI 192.168.204.142@tcp [ 2102.784803] Key type .llcrypt unregistered [ 2102.787561] Key type ._llcrypt unregistered [ 2132.555780] Key type ._llcrypt registered [ 2132.567047] Key type .llcrypt registered [ 2132.739069] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2133.976193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2134.036086] alg: No test for adler32 (adler32-zlib) [ 2135.264872] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 2135.548623] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2137.263187] Key type lgssc registered [ 2138.695716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2147.837161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2147.865254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2149.490347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2154.361330] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2154.466607] LustreError: 79594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.500342] LustreError: 79594:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2159.589921] LustreError: 79595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2164.550092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2164.710780] LustreError: 79594:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.357179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2171.472625] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2180.616159] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2187.951025] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2195.979941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2196.754835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.830206] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2212.096380] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2229.154708] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2230.416409] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2232.352645] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2232.357925] Lustre: Skipped 1 previous similar message [ 2234.406340] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2234.414932] Lustre: Skipped 1 previous similar message [ 2238.738490] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2238.749366] Lustre: Skipped 4 previous similar messages [ 2247.508542] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2247.516854] Lustre: Skipped 8 previous similar messages [ 2264.140355] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2264.145266] Lustre: Skipped 14 previous similar messages [ 2296.385159] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2296.390417] Lustre: Skipped 32 previous similar messages [ 2335.014276] Lustre: 87720:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2337.791605] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:05:58 (1788660358) [ 2339.080495] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2340.922645] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:06:01 (1788660361) [ 2347.491836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2347.498757] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2347.508977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2352.612836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2352.623641] Lustre: Skipped 1 previous similar message [ 2357.734375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2357.754119] Lustre: Skipped 1 previous similar message [ 2359.264543] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2359.560561] Lustre: server umount lustre-OST0000 complete [ 2364.902715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2364.913961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2364.928590] Lustre: Skipped 1 previous similar message [ 2364.935240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2367.973428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2373.090304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2373.111922] Lustre: Skipped 3 previous similar messages [ 2378.208573] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2378.457224] Lustre: server umount lustre-MDT0000 complete [ 2382.676482] LustreError: 79573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788660405 with bad export cookie 12041769756186231550 [ 2382.684181] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2382.700172] LustreError: 79573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2383.086884] Lustre: server umount lustre-MDT0001 complete [ 2394.790270] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2397.866301] Key type lgssc unregistered [ 2398.185409] LNet: 88970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2398.194466] LNetError: 88970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2398.208869] LNet: Removed LNI 192.168.204.142@tcp [ 2399.337237] Key type .llcrypt unregistered [ 2399.340549] Key type ._llcrypt unregistered [ 2404.639721] Key type ._llcrypt registered [ 2404.642560] Key type .llcrypt registered [ 2404.785219] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2409.997488] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules [ 2410.763934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2410.870938] alg: No test for adler32 (adler32-zlib) [ 2412.053434] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 2412.395386] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2414.068802] Key type lgssc registered [ 2415.914466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2417.241498] LustreError: 90168:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2436.442373] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2439.741482] LustreError: 90168:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2447.641401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2447.671301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2449.296471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2454.558189] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2459.621883] LustreError: 91188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2459.647671] LustreError: 91188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2464.737856] LustreError: 91189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2466.754272] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:08:06 (1788660486) [ 2469.859594] LustreError: 91188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2474.050995] LustreError: 91219:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2474.069889] LustreError: 91219:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2474.081434] LustreError: 91219:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 2474.681295] Lustre: server umount lustre-MDT0000 complete [ 2490.795667] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2493.841286] Key type lgssc unregistered [ 2494.183873] LNet: 92713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2494.189115] LNetError: 92713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2494.203443] LNet: Removed LNI 192.168.204.142@tcp [ 2495.153104] Key type .llcrypt unregistered [ 2495.156578] Key type ._llcrypt unregistered [ 2513.795611] Key type ._llcrypt registered [ 2513.800138] Key type .llcrypt registered [ 2513.936757] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2515.580956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2515.631317] alg: No test for adler32 (adler32-zlib) [ 2516.776520] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 2517.051842] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2518.735458] Key type lgssc registered [ 2520.331879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2529.772937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2529.816523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.700450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2538.412367] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2542.049780] LustreError: 93931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.067534] LustreError: 93931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2547.171606] LustreError: 93930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.789390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2549.780296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2554.663561] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2563.520472] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2569.390345] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2576.298901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2576.751747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2583.632379] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2593.793906] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2660.262499] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:11:20 (1788660680) [ 2662.635933] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2665.576932] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:11:25 (1788660685) [ 2672.098222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2672.106408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.130854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2673.120627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2673.121904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2673.147242] Lustre: Skipped 1 previous similar message [ 2673.174343] Lustre: Skipped 2 previous similar messages [ 2677.272490] Lustre: server umount lustre-MDT0000 complete [ 2678.243595] LustreError: 93930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.264413] LustreError: 93930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2681.905371] LustreError: 95782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788660704 with bad export cookie 18302331496032463121 [ 2681.931827] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2682.531581] Lustre: server umount lustre-MDT0001 complete [ 2696.658730] Lustre: server umount lustre-OST0000 complete [ 2708.658326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2709.087446] LustreError: 98252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2709.287921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.596202] LustreError: 98253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2715.102884] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2723.816322] LustreError: 98253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.823920] LustreError: 98253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2725.345905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2726.338252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2730.859968] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2739.023609] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2745.094944] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2757.424341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2757.673748] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2757.682646] LustreError: 98240:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2757.711915] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2757.727029] LustreError: 100336:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2757.741381] LustreError: 100336:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2757.750738] LustreError: 100336:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2757.819858] Lustre: server umount lustre-OST0000 complete [ 2757.823854] LustreError: 100336:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2766.999962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2767.262696] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2767.274727] LustreError: 98241:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2767.283815] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2767.299439] LustreError: 100798:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2767.316244] LustreError: 100798:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2767.323531] LustreError: 100798:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2767.439917] Lustre: server umount lustre-OST0000 complete [ 2767.445709] LustreError: 100798:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2777.463965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2777.655300] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2777.661547] Lustre: Found index 0 for lustre-OST0000, updating log [ 2777.670358] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2777.740564] Lustre: lustre-OST0000: new disk, initializing [ 2777.744098] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2777.749205] Lustre: 101331:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2778.053410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2784.589486] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2788.658861] Lustre: Failing over lustre-OST0000 [ 2788.751961] Lustre: server umount lustre-OST0000 complete [ 2792.420300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2792.430627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2792.441938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2794.472286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2794.475059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2794.496727] Lustre: Skipped 1 previous similar message [ 2800.609897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2800.627347] Lustre: Skipped 3 previous similar messages [ 2806.239292] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2806.553367] Lustre: server umount lustre-MDT0000 complete [ 2810.853833] LustreError: 98252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.884797] LustreError: 98252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2812.153873] LustreError: 100355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788660835 with bad export cookie 18302331496032465872 [ 2812.184481] LustreError: 100355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2812.186431] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2812.610449] Lustre: server umount lustre-MDT0001 complete [ 2823.889516] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:14:04 (1788660844) [ 2842.710283] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2853.763536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2854.342857] LustreError: 103678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2854.631868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2859.807958] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2868.672526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2874.449572] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2877.505935] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2884.840968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2885.662613] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2892.842959] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2893.862797] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2893.890928] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2893.911788] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2898.919721] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2898.940389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2899.947273] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2899.979655] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2899.982327] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2899.996885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2902.438652] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2902.490656] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2902.624924] Lustre: lustre-OST0001: new disk, initializing [ 2902.632334] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2902.637462] Lustre: 105999:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2902.961533] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2902.975680] Lustre: Skipped 2 previous similar messages [ 2910.909974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2910.917520] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2911.111628] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2912.492746] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2922.780962] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2931.482885] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2936.295440] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2936.302860] LustreError: Skipped 1 previous similar message [ 2936.308083] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2936.323105] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2936.329578] Lustre: Skipped 2 previous similar messages [ 2943.025176] Lustre: server umount lustre-MDT0001 complete [ 2945.086511] LustreError: 104419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2945.132028] LustreError: 104419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2955.103875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2955.231787] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2955.237329] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2955.315782] Lustre: 103666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2957.388164] LustreError: 107542:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2957.400730] LustreError: 107542:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2957.408921] LustreError: 103670:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2957.494977] LustreError: 107553:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.142@tcp: cfg command failed: rc = -17 [ 2957.501261] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2957.501261] [ 2961.915673] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2962.575634] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2974.655360] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 22:16:35 (1788660995) [ 2996.814285] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3018.556335] Lustre: 109614:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3046.242366] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3051.040411] Lustre: 110750:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3145.183835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3145.199718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.225323] Lustre: Skipped 2 previous similar messages [ 3145.230700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3145.235454] Lustre: Skipped 7 previous similar messages [ 3150.021514] Lustre: server umount lustre-MDT0000 complete [ 3151.328565] LustreError: 103675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3151.360288] LustreError: 103675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 3154.268799] LustreError: 103659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788661177 with bad export cookie 18302331496032466803 [ 3154.271340] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3154.299298] LustreError: 103659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3154.682117] Lustre: server umount lustre-MDT0001 complete [ 3168.248327] Lustre: server umount lustre-OST0000 complete [ 3183.090416] Lustre: server umount lustre-OST0001 complete [ 3189.398264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3201.461583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3210.577951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3221.913411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3248.692705] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3265.425325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.440577] Lustre: lustre-MDT0000: reset Object Index mappings [ 3265.977626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3265.981332] Lustre: Skipped 1 previous similar message [ 3271.804996] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3282.834780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3282.882594] Lustre: lustre-MDT0001: reset Object Index mappings [ 3288.912169] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3292.683588] Lustre: 115853:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3301.048233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3308.377677] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3309.068271] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3311.098956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:386 to 0x280000401:418) [ 3317.883464] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3323.920305] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3323.933535] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3325.207739] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3333.891646] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3337.609387] Lustre: 117730:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3393.640615] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:23:34 (1788661414) [ 3421.153649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3421.168481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3421.182055] Lustre: Skipped 6 previous similar messages [ 3421.198277] Lustre: Skipped 7 previous similar messages [ 3425.172597] Lustre: server umount lustre-MDT0000 complete [ 3426.278593] LustreError: 114705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3426.299635] LustreError: 114705:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 3429.688906] LustreError: 114685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788661452 with bad export cookie 18302331496032513892 [ 3429.689306] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3429.700696] LustreError: 114685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3436.142146] Lustre: server umount lustre-MDT0001 complete [ 3446.720955] Lustre: server umount lustre-OST0000 complete [ 3465.183364] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3465.469562] Lustre: server umount lustre-OST0001 complete [ 3473.197411] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 3483.862644] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3524.783975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3525.168120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3525.212328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3525.326819] Lustre: lustre-MDT0000: new disk, initializing [ 3525.462809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3525.480092] Lustre: Skipped 3 previous similar messages [ 3525.514835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3530.992804] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3539.942737] Lustre: server umount lustre-MDT0000 complete [ 3551.072814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3551.391171] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3551.418694] Lustre: 122008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3551.429821] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3557.029980] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3563.869215] Lustre: server umount lustre-MDT0000 complete [ 3573.800412] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:26:34 (1788661594) [ 3575.607468] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3578.146406] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:26:38 (1788661598) [ 3602.000584] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3614.793421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3620.756930] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3631.111535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3631.239377] Lustre: 124075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3631.304851] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3631.309681] Lustre: Skipped 1 previous similar message [ 3631.356225] Lustre: lustre-MDT0001: new disk, initializing [ 3631.475357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3631.492874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3638.261112] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3643.870344] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3651.375344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3651.788710] Lustre: lustre-OST0000: new disk, initializing [ 3651.797738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3651.822279] Lustre: 125653:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3653.804302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3653.821261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3654.036558] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3660.187316] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3671.612528] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3671.777405] Lustre: lustre-OST0001: new disk, initializing [ 3671.780610] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3671.786512] Lustre: 126525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3673.767185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3673.780969] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3673.908242] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3679.240694] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3689.183255] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3692.432679] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4129.547357] Lustre: 132948:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4129.557098] Lustre: 132948:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4129.565701] Lustre: 132948:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4129.574730] Lustre: 132948:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4129.585357] Lustre: 132948:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4129.598304] Lustre: 132948:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4450.555582] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:41:11 (1788662471) [ 4465.636270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4465.638277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4465.657201] Lustre: Skipped 4 previous similar messages [ 4465.661475] Lustre: Skipped 7 previous similar messages [ 4470.775332] LustreError: 127332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4470.784922] LustreError: 127332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4471.019555] Lustre: server umount lustre-MDT0000 complete [ 4475.947337] LustreError: 132980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788662498 with bad export cookie 18302331496032577557 [ 4475.958150] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4475.975511] LustreError: 132980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4481.008066] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4481.013142] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4481.037676] Lustre: Skipped 2 previous similar messages [ 4481.052894] Lustre: Skipped 1 previous similar message [ 4482.611524] Lustre: server umount lustre-MDT0001 complete [ 4493.938974] Lustre: server umount lustre-OST0000 complete [ 4512.223706] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4512.506606] Lustre: server umount lustre-OST0001 complete [ 4520.478650] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 4532.165410] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4569.429533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4569.702831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4569.750245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4569.809629] Lustre: lustre-MDT0000: new disk, initializing [ 4569.928703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4569.935295] Lustre: Skipped 5 previous similar messages [ 4569.956667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4574.446504] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4585.533332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4585.661328] Lustre: 136021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4585.717373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4585.722319] Lustre: Skipped 1 previous similar message [ 4585.788580] Lustre: lustre-MDT0001: new disk, initializing [ 4586.000535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4586.013411] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4590.315995] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4601.015970] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4606.878770] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4614.196361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4614.484706] Lustre: lustre-OST0000: new disk, initializing [ 4614.488429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4614.502705] Lustre: 137989:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4614.615151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4614.619946] Lustre: Skipped 1 previous similar message [ 4616.392429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4616.401964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4616.518327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4621.769469] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4633.364248] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4645.189774] Lustre: lustre-MDD0000: changelog on [ 4648.194105] Lustre: lustre-MDD0001: changelog on [ 6811.803458] Lustre: lustre-MDD0001: changelog off [ 6813.740749] Lustre: 136026:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6814.682394] Lustre: lustre-MDD0000: changelog off [ 6823.398274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6823.411350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6823.452088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6823.469633] Lustre: Skipped 1 previous similar message [ 6827.454818] Lustre: server umount lustre-OST0000 complete [ 6833.639983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6833.652571] Lustre: Skipped 2 previous similar messages [ 6833.652952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6833.671725] Lustre: Skipped 1 previous similar message [ 6838.756197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6838.762730] Lustre: Skipped 1 previous similar message [ 6845.407265] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6845.694207] Lustre: server umount lustre-MDT0000 complete [ 6849.002665] LustreError: 136028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6849.018029] LustreError: 136028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6849.626436] LustreError: 137988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788664872 with bad export cookie 18302331496033405195 [ 6849.631837] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6849.644188] LustreError: 137988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6850.077280] Lustre: server umount lustre-MDT0001 complete [ 6864.113062] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6867.521607] Key type lgssc unregistered [ 6867.735952] LNet: 147174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6867.755144] LNetError: 147174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6867.793287] LNet: Removed LNI 192.168.204.142@tcp [ 6868.652200] Key type .llcrypt unregistered [ 6868.653957] Key type ._llcrypt unregistered [ 6890.665422] Key type ._llcrypt registered [ 6890.674080] Key type .llcrypt registered [ 6890.822155] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 6906.807319] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6908.161263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6908.197804] alg: No test for adler32 (adler32-zlib) [ 6909.334854] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 6909.658838] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6911.384189] Key type lgssc registered [ 6912.317759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6949.293511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6949.319884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6950.625727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6950.658491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6950.753481] Lustre: lustre-MDT0000: new disk, initializing [ 6950.830490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6950.846689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6955.470673] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6965.966232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.119865] Lustre: 150375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6966.222750] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6966.228823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6966.238674] Lustre: Skipped 1 previous similar message [ 6966.331557] Lustre: lustre-MDT0001: new disk, initializing [ 6966.457060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6966.495146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6966.505924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6972.051479] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6983.428272] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6990.202532] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6997.444060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6997.803638] Lustre: lustre-OST0000: new disk, initializing [ 6997.809434] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6997.818571] Lustre: 152344:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6997.907061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7004.757077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7004.770664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7004.881835] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7005.136650] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7015.241558] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7022.406470] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_module obdecho/obdecho [ 7024.418943] LustreError: 153348:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7024.428701] LustreError: 153348:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7026.524451] LustreError: 153445:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7040.997673] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7040.998898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7041.010127] Lustre: Skipped 2 previous similar messages [ 7041.031021] Lustre: Skipped 2 previous similar messages [ 7045.411160] Lustre: server umount lustre-MDT0000 complete [ 7050.554321] LustreError: 151172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665073 with bad export cookie 18251463718072571082 [ 7050.558489] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7050.581257] LustreError: 151172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7051.105600] Lustre: server umount lustre-MDT0001 complete [ 7066.336782] Lustre: server umount lustre-OST0000 complete [ 7090.940767] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 7103.292614] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7142.682766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7142.977986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7143.017702] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7143.142107] Lustre: lustre-MDT0000: new disk, initializing [ 7143.253223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7143.290252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7147.971418] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7160.034983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7160.155140] Lustre: 157166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7160.221236] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7160.228029] Lustre: Skipped 1 previous similar message [ 7160.288294] Lustre: lustre-MDT0001: new disk, initializing [ 7160.360550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7160.384388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7160.388941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7164.107111] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7177.581359] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7184.984985] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7193.763974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7194.082729] Lustre: lustre-OST0000: new disk, initializing [ 7194.091193] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7194.106775] Lustre: 159132:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7194.278994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7202.871704] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7203.928370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7203.950749] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7204.112901] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7214.253224] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7219.594522] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7219.890765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7224.459358] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7224.685186] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7229.922122] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7229.931340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7229.933427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7229.940810] Lustre: Skipped 1 previous similar message [ 7232.987089] Lustre: server umount lustre-OST0000 complete [ 7240.161959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7240.171288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7240.176448] Lustre: Skipped 1 previous similar message [ 7240.184615] Lustre: Skipped 1 previous similar message [ 7243.515544] Lustre: server umount lustre-MDT0000 complete [ 7245.282199] LustreError: 157177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7245.313661] LustreError: 157177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7247.023563] LustreError: 157158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665269 with bad export cookie 18251463718072572041 [ 7247.034466] LustreError: 157158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7247.035653] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7247.315929] Lustre: server umount lustre-MDT0001 complete [ 7250.864426] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:27:51 (1788665271) [ 7252.351497] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7254.341691] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:27:54 (1788665274) [ 7261.916742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7262.289818] LustreError: 161547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7262.341737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7267.002151] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7267.814548] LustreError: 161548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7272.930660] LustreError: 161547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7277.138714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7281.841360] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7290.299611] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7297.336460] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7304.550123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7304.978599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7304.986348] Lustre: Skipped 1 previous similar message [ 7311.371414] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7319.152405] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7332.842389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7332.851641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7332.867785] Lustre: Skipped 2 previous similar messages [ 7332.878798] Lustre: Skipped 2 previous similar messages [ 7338.467096] Lustre: server umount lustre-MDT0000 complete [ 7341.113808] LustreError: 161543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7342.559185] LustreError: 161527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665365 with bad export cookie 18251463718072572923 [ 7342.563671] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7342.597032] LustreError: 161527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7343.074331] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7343.086941] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7343.091379] Lustre: Skipped 4 previous similar messages [ 7348.971728] Lustre: server umount lustre-MDT0001 complete [ 7354.933431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7366.549597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7366.738390] LustreError: 165410:0:(llog.c:1655:llog_backup()) MGC192.168.204.142@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7366.744814] Lustre: 165410:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.142@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7366.775907] LustreError: 165415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7366.798153] LustreError: 165415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7383.010720] LustreError: 165417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7383.033181] LustreError: 165417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7418.336716] LustreError: 165415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7418.352471] LustreError: 165415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 7428.063587] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7428.076100] Lustre: 165410:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.142@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7469.048393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7473.769324] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7474.151942] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7481.963155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7483.445892] LustreError: 165417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7483.467738] LustreError: 165417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 7487.472686] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfd4a398d1aa57531 [ 7487.489959] Lustre: MGC192.168.204.142@tcp: Connection restored to 0@lo (at 0@lo) [ 7492.535237] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7493.121325] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7500.587591] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7502.928401] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7509.369933] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7511.003462] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7532.908687] Lustre: server umount lustre-MDT0000 complete [ 7534.053575] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7534.061730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7534.068301] Lustre: Skipped 2 previous similar messages [ 7534.084317] LustreError: Skipped 1 previous similar message [ 7536.797437] LustreError: 161528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665559 with bad export cookie 18251463718072579377 [ 7536.800949] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7536.806763] LustreError: 161528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7537.186820] Lustre: server umount lustre-MDT0001 complete [ 7550.499414] Lustre: server umount lustre-OST0000 complete [ 7573.869823] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 7583.849579] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7626.327635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7626.673834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7626.739407] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7626.900478] Lustre: lustre-MDT0000: new disk, initializing [ 7627.028306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7627.034596] Lustre: Skipped 1 previous similar message [ 7627.052396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7632.378819] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7644.121429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7644.249311] Lustre: 170518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7644.557156] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7648.989251] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7658.382260] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7664.836786] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7672.728763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7672.995551] Lustre: lustre-OST0000: new disk, initializing [ 7672.999348] Lustre: Skipped 1 previous similar message [ 7673.003569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7673.009883] Lustre: Skipped 2 previous similar messages [ 7673.014559] Lustre: 172487:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7679.047926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7679.071249] Lustre: Skipped 1 previous similar message [ 7679.077630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7679.185705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7679.748247] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7690.076180] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7694.868422] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7695.082423] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7700.094879] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7700.385104] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7705.056055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7705.066261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7705.071301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7705.079281] Lustre: Skipped 1 previous similar message [ 7705.089181] Lustre: Skipped 2 previous similar messages [ 7708.848598] Lustre: server umount lustre-OST0000 complete [ 7719.531997] Lustre: server umount lustre-MDT0000 complete [ 7720.417095] LustreError: 170529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7720.429682] LustreError: 170529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7723.170981] LustreError: 170509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665746 with bad export cookie 18251463718072596198 [ 7723.173684] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7723.180202] LustreError: 170509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7723.499443] Lustre: server umount lustre-MDT0001 complete [ 7734.481356] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:35:54 (1788665754) [ 7743.864393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7748.823914] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7760.765520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7766.657096] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7777.062643] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7783.727342] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7790.435074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7797.569737] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7805.594367] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7829.472479] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:37:29 (1788665849) [ 7835.107960] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7835.110055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7835.117391] LustreError: Skipped 1 previous similar message [ 7835.118977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7835.118982] Lustre: Skipped 3 previous similar messages [ 7835.149778] Lustre: Skipped 3 previous similar messages [ 7840.622595] Lustre: server umount lustre-OST0000 complete [ 7850.901948] Lustre: server umount lustre-MDT0000 complete [ 7855.055875] LustreError: 174881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788665877 with bad export cookie 18251463718072597087 [ 7855.066107] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7855.075261] LustreError: 174881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7855.526332] Lustre: server umount lustre-MDT0001 complete [ 7866.610472] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 7869.651493] Key type lgssc unregistered [ 7870.028047] LNet: 178651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7870.047789] LNetError: 178651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7871.082525] LNet: Removed LNI 192.168.204.142@tcp [ 7872.109893] Key type .llcrypt unregistered [ 7872.113856] Key type ._llcrypt unregistered [ 7891.291943] Key type ._llcrypt registered [ 7891.295587] Key type .llcrypt registered [ 7891.393642] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7892.334729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7892.375131] alg: No test for adler32 (adler32-zlib) [ 7893.384200] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 7893.598412] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 7895.311197] Key type lgssc registered [ 7896.733731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7906.147504] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7906.165681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7921.759401] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7926.879875] LustreError: 179891:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.142@tcp: failed processing log, type 4: rc = -110 [ 7932.000201] LustreError: 179891:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7932.035228] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7932.040874] LustreError: MGC192.168.204.142@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7932.059513] LustreError: 179891:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7932.074131] LustreError: 179891:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 7932.082304] LustreError: 179891:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7932.227441] Lustre: server umount lustre-OST0000 complete [ 7932.231934] LustreError: 179891:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7943.660393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7945.378415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7950.832533] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7955.939645] LustreError: 180390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7955.959765] LustreError: 180390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7960.303970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7960.725349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7966.440610] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7975.028685] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7981.433107] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7989.612321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7989.892044] LustreError: 182207:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7989.904125] Lustre: 182207:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.142@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7990.124158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7997.875275] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8006.171396] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8010.731521] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8010.740054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8010.761470] Lustre: Skipped 1 previous similar message [ 8010.788082] Lustre: Skipped 2 previous similar messages [ 8014.841809] Lustre: server umount lustre-MDT0000 complete [ 8015.849381] LustreError: 180391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8015.878781] LustreError: 180391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8019.789663] LustreError: 182235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666042 with bad export cookie 18338630011450698950 [ 8019.798850] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8019.806060] LustreError: 182235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8020.297996] Lustre: server umount lustre-MDT0001 complete [ 8034.946639] Lustre: server umount lustre-OST0000 complete [ 8045.907498] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8061.837320] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8107.561915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8107.812402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8107.882699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8108.065509] Lustre: lustre-MDT0000: new disk, initializing [ 8108.149666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8108.165871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8114.436678] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8126.364627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8126.559901] Lustre: 185748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8126.624694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8126.629668] Lustre: Skipped 1 previous similar message [ 8126.756537] Lustre: lustre-MDT0001: new disk, initializing [ 8126.964256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8127.049129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8127.064164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8130.871627] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8141.900413] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8149.386968] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8157.574249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8157.978886] Lustre: lustre-OST0000: new disk, initializing [ 8157.984919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8158.000197] Lustre: 187718:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8158.145876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8164.824225] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8166.484996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8166.501100] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8166.600798] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8175.217689] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8181.537389] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8181.784160] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8187.708911] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8188.006928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8192.481069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8192.483165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8192.494720] LustreError: Skipped 1 previous similar message [ 8192.500763] Lustre: Skipped 1 previous similar message [ 8192.501740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8192.519409] Lustre: Skipped 1 previous similar message [ 8197.607082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8197.624785] Lustre: Skipped 1 previous similar message [ 8202.742706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8202.761175] Lustre: Skipped 1 previous similar message [ 8204.255167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8204.542753] Lustre: server umount lustre-OST0000 complete [ 8208.865012] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8208.870632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.888431] Lustre: Skipped 1 previous similar message [ 8208.895829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8212.967098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8215.073157] Lustre: server umount lustre-MDT0000 complete [ 8218.082732] LustreError: 186869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8218.122884] LustreError: 186869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8219.043109] LustreError: 187728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666241 with bad export cookie 18338630011450699671 [ 8219.049550] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8219.063809] LustreError: 187728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8219.377278] Lustre: server umount lustre-MDT0001 complete [ 8230.317501] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:44:10 (1788666250) [ 8251.140576] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 8254.257193] Key type lgssc unregistered [ 8254.627510] LNet: 190665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8254.642290] LNetError: 190665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8254.663312] LNet: Removed LNI 192.168.204.142@tcp [ 8255.918692] Key type .llcrypt unregistered [ 8255.920729] Key type ._llcrypt unregistered [ 8277.668288] Key type ._llcrypt registered [ 8277.671291] Key type .llcrypt registered [ 8277.768600] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8278.576271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8278.593588] alg: No test for adler32 (adler32-zlib) [ 8279.975404] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 8280.360735] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 8282.039350] Key type lgssc registered [ 8283.676738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8293.048862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8293.071733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8294.460885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8299.816273] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8304.612348] LustreError: 191881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8304.630432] LustreError: 191881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8309.736699] LustreError: 191880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8310.278330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8310.902865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8316.362969] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8325.242910] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8331.225751] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8340.271459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8340.439562] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8340.443228] LustreError: 193743:0:(llog.c:1696:llog_backup()) MGC192.168.204.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8340.456128] Lustre: 193743:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.142@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8340.639587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8349.844084] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8360.610742] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8366.562889] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8366.568453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8366.571964] Lustre: Skipped 2 previous similar messages [ 8366.596428] Lustre: Skipped 2 previous similar messages [ 8368.847860] Lustre: server umount lustre-MDT0000 complete [ 8371.683515] LustreError: 191880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8371.702199] LustreError: 191880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8373.445708] LustreError: 194505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666396 with bad export cookie 18331470385517266274 [ 8373.450083] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8373.464186] LustreError: 194505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8373.736944] Lustre: server umount lustre-MDT0001 complete [ 8387.728351] Lustre: server umount lustre-OST0000 complete [ 8398.868945] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8412.109924] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8455.882343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8456.253316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8456.299492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8456.438979] Lustre: lustre-MDT0000: new disk, initializing [ 8456.624852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8456.656654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8462.421250] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8476.119089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8476.341793] Lustre: 197270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8476.425663] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8476.432735] Lustre: Skipped 1 previous similar message [ 8476.437019] LustreError: 198057:0:(llog.c:1696:llog_backup()) MGC192.168.204.142@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8476.445370] LustreError: 198057:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8476.450889] Lustre: 198057:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.142@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8476.472449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8476.476758] Lustre: Skipped 1 previous similar message [ 8476.634970] Lustre: lustre-MDT0001: new disk, initializing [ 8476.788609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8476.848391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8476.867533] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8482.774526] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8494.860761] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8501.285567] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8508.454772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8508.672279] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8508.677200] Lustre: Skipped 1 previous similar message [ 8508.681481] LustreError: 199219:0:(llog.c:1696:llog_backup()) MGC192.168.204.142@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8508.687333] LustreError: 199219:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8508.692142] Lustre: 199219:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.142@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8508.699410] Lustre: 199219:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8508.715778] Lustre: lustre-OST0000: new disk, initializing [ 8508.718918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8508.723560] Lustre: 199242:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8508.820809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8515.884524] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8518.246677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8518.257320] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8518.411332] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8526.069493] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8531.189792] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8531.438980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8535.971830] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8536.138697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8539.108834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8539.115286] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8539.116869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8539.122479] Lustre: Skipped 1 previous similar message [ 8539.142257] LustreError: Skipped 1 previous similar message [ 8544.229193] LustreError: 199227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8544.304226] Lustre: server umount lustre-OST0000 complete [ 8548.321866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8548.337485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8548.357203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8548.368450] Lustre: Skipped 1 previous similar message [ 8554.299680] Lustre: server umount lustre-MDT0000 complete [ 8554.469224] LustreError: 198644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8554.488433] LustreError: 198644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8558.303369] LustreError: 197263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666581 with bad export cookie 18331470385517266988 [ 8558.329294] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8558.334649] LustreError: 197263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8558.821777] Lustre: server umount lustre-MDT0001 complete [ 8569.449064] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:49:49 (1788666589) [ 8588.152732] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8600.401084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8600.812699] LustreError: 202370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8600.855480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8605.821223] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8615.068653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8619.965274] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8623.046469] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8631.136811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8638.393606] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8647.423861] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8647.656277] Lustre: lustre-OST0001: new disk, initializing [ 8647.660487] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8647.666593] Lustre: 204678:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8647.764145] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8647.769640] Lustre: Skipped 2 previous similar messages [ 8654.074395] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8656.983225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8657.008778] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8657.197054] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8664.211436] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8670.585363] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8672.559923] Lustre: DEBUG MARKER: ADD OST3 [ 8677.940710] loop: module loaded [ 8678.042658] loop0: detected capacity change from 0 to 204800000 [ 8678.302123] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8678.321363] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8678.365128] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8686.251639] Lustre: DEBUG MARKER: STOP OST3 [ 8686.551306] loop0: detected capacity change from 0 to 204800000 [ 8693.736254] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8693.947867] Lustre: lustre-OST0003: new disk, initializing [ 8693.953358] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8693.966207] Lustre: 206721:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8695.772495] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8695.910102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8695.920174] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8695.970305] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8697.799740] Lustre: Failing over lustre-OST0003 [ 8697.823136] LustreError: 206839:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8698.179485] Lustre: server umount lustre-OST0003 complete [ 8700.904727] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8700.906738] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8700.919803] Lustre: Skipped 1 previous similar message [ 8700.931714] LustreError: 203855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8700.960649] LustreError: 203855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8702.459463] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8710.991616] Lustre: DEBUG MARKER: START OST3 again [ 8717.583648] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8717.886427] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8717.890726] Lustre: Skipped 1 previous similar message [ 8717.909159] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8719.466387] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8719.560235] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8719.561048] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8724.614331] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8746.697238] Lustre: Failing over lustre-OST0003 [ 8746.859853] Lustre: server umount lustre-OST0003 complete [ 8748.531106] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8748.534702] LustreError: 207614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8748.539724] Lustre: Skipped 2 previous similar messages [ 8748.554601] LustreError: 207614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8768.992091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8768.995414] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8769.005246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8769.005254] Lustre: Skipped 2 previous similar messages [ 8769.048944] Lustre: Skipped 3 previous similar messages [ 8772.092762] Lustre: server umount lustre-MDT0000 complete [ 8776.018975] LustreError: 205446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666798 with bad export cookie 18331470385517267870 [ 8776.030191] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8776.034321] LustreError: 205446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8779.236996] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8779.246071] Lustre: Skipped 4 previous similar messages [ 8782.598830] Lustre: server umount lustre-MDT0001 complete [ 8786.582927] Lustre: server umount lustre-OST0000 complete [ 8790.953891] Lustre: server umount lustre-OST0001 complete [ 8798.874554] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8808.385955] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8849.373230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8849.692871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8849.749333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8849.837625] Lustre: lustre-MDT0000: new disk, initializing [ 8849.922042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8849.945262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8855.386771] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8868.435715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8868.603435] Lustre: 212140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8868.691220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8868.694937] Lustre: Skipped 1 previous similar message [ 8868.799861] Lustre: lustre-MDT0001: new disk, initializing [ 8868.975096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8868.995644] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8874.311384] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8885.384282] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8891.571867] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8898.513604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8898.798992] Lustre: 214111:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8900.713216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8900.818770] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8904.363783] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 8913.361084] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8918.873552] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8919.096267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8924.712424] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8924.974900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8931.813394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8931.815615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8931.822595] LustreError: Skipped 1 previous similar message [ 8931.848574] Lustre: Skipped 2 previous similar messages [ 8931.858925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8931.881745] Lustre: Skipped 3 previous similar messages [ 8933.780887] Lustre: server umount lustre-OST0000 complete [ 8940.512345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8951.270254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8951.280434] Lustre: Skipped 6 previous similar messages [ 8952.287121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8952.629558] Lustre: server umount lustre-MDT0000 complete [ 8956.385361] LustreError: 212153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8956.428863] LustreError: 212153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 8957.104144] LustreError: 212132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788666979 with bad export cookie 18331470385517270166 [ 8957.106877] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8957.113844] LustreError: 212132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8957.561794] Lustre: server umount lustre-MDT0001 complete [ 8962.058276] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 23:56:22 (1788666982) [ 8981.803349] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8996.582925] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9043.515496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9043.834559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9043.906583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9043.911131] Lustre: Skipped 1 previous similar message [ 9044.002776] Lustre: lustre-MDT0000: new disk, initializing [ 9044.005957] Lustre: Skipped 1 previous similar message [ 9044.115966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9044.124488] Lustre: Skipped 1 previous similar message [ 9049.041286] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9061.165544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9061.287355] Lustre: 218739:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9061.588783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9067.351382] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9078.868237] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9085.162347] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9094.308026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9094.581715] Lustre: 220709:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9096.445641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9096.714936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9103.311222] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9113.565568] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9118.868700] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9119.335791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9126.321963] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9126.690596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9132.515226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9132.518096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9132.520112] Lustre: Skipped 3 previous similar messages [ 9132.526703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9132.532110] Lustre: Skipped 1 previous similar message [ 9143.272617] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9143.622414] Lustre: server umount lustre-OST0000 complete [ 9154.577136] Lustre: server umount lustre-MDT0000 complete [ 9158.113085] LustreError: 218746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9158.137851] LustreError: 218746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9158.716296] LustreError: 220708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667181 with bad export cookie 18331470385517271048 [ 9158.734387] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9158.744075] LustreError: 220708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9159.017566] Lustre: server umount lustre-MDT0001 complete [ 9168.178947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9168.584823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9168.590277] Lustre: Skipped 5 previous similar messages [ 9174.133403] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9183.454085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9188.926244] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9197.814841] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9202.874826] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9210.024651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9217.045865] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9225.871729] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9239.023380] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9239.026266] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9239.047854] Lustre: Skipped 2 previous similar messages [ 9239.048570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9239.048575] Lustre: Skipped 7 previous similar messages [ 9239.095883] LustreError: Skipped 2 previous similar messages [ 9242.734930] Lustre: server umount lustre-OST0000 complete [ 9253.369638] Lustre: server umount lustre-MDT0000 complete [ 9257.841545] LustreError: 222911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667280 with bad export cookie 18331470385517271930 [ 9257.852571] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9257.864291] LustreError: 222911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9258.391931] Lustre: server umount lustre-MDT0001 complete [ 9365.731537] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 9369.102500] Key type lgssc unregistered [ 9369.466889] LNet: 226604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9369.477647] LNetError: 226604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9369.511963] LNet: Removed LNI 192.168.204.142@tcp [ 9370.513178] Key type .llcrypt unregistered [ 9370.517168] Key type ._llcrypt unregistered [ 9397.307595] Key type ._llcrypt registered [ 9397.310846] Key type .llcrypt registered [ 9397.400599] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 9416.604309] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9417.817222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9417.857375] alg: No test for adler32 (adler32-zlib) [ 9419.280233] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [ 9419.573904] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 9421.311169] Key type lgssc registered [ 9422.596235] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9464.569857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9464.581934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9465.929167] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9465.957121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9466.084498] Lustre: lustre-MDT0000: new disk, initializing [ 9466.216985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9466.253714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9472.022230] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9484.534813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9484.704258] Lustre: 229957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9484.771509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9484.775330] Lustre: Skipped 1 previous similar message [ 9484.850247] Lustre: lustre-MDT0001: new disk, initializing [ 9485.033296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9485.106135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9485.126266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9490.546717] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9501.716501] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9510.076338] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9518.560898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9518.933796] Lustre: lustre-OST0000: new disk, initializing [ 9518.939972] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9518.950383] Lustre: 231928:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9519.106687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9526.098563] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9526.916838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9526.957189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9527.087567] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9536.527795] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9542.472653] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9542.740135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9548.596376] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9548.828389] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9552.865578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9552.871341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9552.880550] LustreError: Skipped 1 previous similar message [ 9552.898232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9552.909945] Lustre: Skipped 1 previous similar message [ 9557.994942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9558.008123] Lustre: Skipped 1 previous similar message [ 9563.111358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9563.129558] Lustre: Skipped 1 previous similar message [ 9565.157276] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9565.393424] Lustre: server umount lustre-OST0000 complete [ 9572.323546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9572.343366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9572.373746] Lustre: Skipped 1 previous similar message [ 9572.385279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9573.349725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9576.921378] Lustre: server umount lustre-MDT0000 complete [ 9582.336369] LustreError: 230752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667605 with bad export cookie 5559560951483393523 [ 9582.344174] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9582.369929] LustreError: 230752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9582.849871] Lustre: server umount lustre-MDT0001 complete [ 9608.377881] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9624.375907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9625.390492] LustreError: 234885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9625.420504] LustreError: 234885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9625.501406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9630.691564] LustreError: 234886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9631.568263] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9635.814480] LustreError: 234885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9640.934516] LustreError: 234886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9641.798046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9642.327828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9647.049681] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9650.521665] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9657.855832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9658.297230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9665.750712] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9675.650306] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9675.929150] Lustre: lustre-OST0001: new disk, initializing [ 9675.932531] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9675.939867] Lustre: 237195:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9676.066739] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9682.085541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9682.124604] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9682.293638] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9683.514339] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9695.303380] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9706.252379] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9722.673827] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:09:03 (1788667743) [ 9728.489450] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9728.495215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9728.535364] Lustre: Skipped 3 previous similar messages [ 9728.552621] Lustre: Skipped 5 previous similar messages [ 9738.720784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9738.740399] Lustre: Skipped 4 previous similar messages [ 9739.231148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9739.437789] Lustre: server umount lustre-MDT0000 complete [ 9743.693073] LustreError: 234866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667766 with bad export cookie 5559560951483394405 [ 9743.697070] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9743.704910] LustreError: 234866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9743.848493] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9743.851656] LustreError: 236392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9743.868355] Lustre: Skipped 1 previous similar message [ 9743.911043] LustreError: 236392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9744.173410] Lustre: server umount lustre-MDT0001 complete [ 9751.065565] Lustre: server umount lustre-OST0000 complete [ 9755.129923] Lustre: server umount lustre-OST0001 complete [ 9762.151507] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 9772.264685] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 9809.628726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9809.839678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9809.875337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9809.970428] Lustre: lustre-MDT0000: new disk, initializing [ 9810.085887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9810.105929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9815.650326] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9827.442823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9827.575096] Lustre: 241142:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9827.631642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9827.638114] Lustre: Skipped 1 previous similar message [ 9827.710792] Lustre: lustre-MDT0001: new disk, initializing [ 9827.832730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9827.853351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9833.610807] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9843.988506] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9850.174921] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9858.716370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9859.151951] Lustre: lustre-OST0000: new disk, initializing [ 9859.158549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9859.177361] Lustre: 243110:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9860.752697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9860.763094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9860.934969] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9865.455536] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9876.311376] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9882.503656] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9882.865168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9890.088285] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9890.365307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9896.929052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9896.934592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9896.940556] LustreError: Skipped 1 previous similar message [ 9896.941639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9896.941645] Lustre: Skipped 4 previous similar messages [ 9896.974557] Lustre: Skipped 1 previous similar message [ 9898.945547] Lustre: server umount lustre-OST0000 complete [ 9904.608459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9909.665187] Lustre: server umount lustre-MDT0000 complete [ 9912.289671] LustreError: 241153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9912.307402] LustreError: 241153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9913.672435] LustreError: 241939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788667936 with bad export cookie 5559560951483396540 [ 9913.675019] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9913.680284] LustreError: 241939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9913.955589] Lustre: server umount lustre-MDT0001 complete [ 9923.812475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9924.293405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9924.305727] Lustre: Skipped 2 previous similar messages [ 9928.642790] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9929.696647] LustreError: 245328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9929.720912] LustreError: 245328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9939.125642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9945.888292] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9955.818689] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9962.845509] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9971.175650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9977.718628] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 9984.548270] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10132.448895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10132.454540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10132.457045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10132.470759] Lustre: Skipped 3 previous similar messages [10132.481747] Lustre: Skipped 5 previous similar messages [10144.224171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10144.370898] Lustre: server umount lustre-OST0000 complete [10149.857722] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10154.784420] Lustre: server umount lustre-MDT0000 complete [10158.049945] LustreError: 245328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10158.076331] LustreError: 245328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10159.212818] LustreError: 245307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668182 with bad export cookie 5559560951483397422 [10159.222571] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10159.236404] LustreError: 245307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10159.761647] Lustre: server umount lustre-MDT0001 complete [10252.580660] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [10256.646966] Key type lgssc unregistered [10257.053505] LNet: 248939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10257.065201] LNetError: 248939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10257.093801] LNet: Removed LNI 192.168.204.142@tcp [10258.209391] Key type .llcrypt unregistered [10258.213704] Key type ._llcrypt unregistered [10287.681807] Key type ._llcrypt registered [10287.683913] Key type .llcrypt registered [10287.779063] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [10304.282896] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10305.740298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10305.785214] alg: No test for adler32 (adler32-zlib) [10306.761067] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [10306.932975] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [10308.607554] Key type lgssc registered [10309.883372] Lustre: Echo OBD driver; http://www.lustre.org/ [10359.614271] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10370.097361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10370.130886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10371.387866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10371.414275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10371.469625] Lustre: lustre-MDT0000: new disk, initializing [10371.523932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10371.545381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10377.039343] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10389.022858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10389.195302] Lustre: 253008:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10389.319220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10389.324490] Lustre: Skipped 1 previous similar message [10389.446127] Lustre: lustre-MDT0001: new disk, initializing [10389.613795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10389.652043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10389.674187] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10395.184617] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10400.843901] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10407.811263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10408.072791] Lustre: lustre-OST0000: new disk, initializing [10408.075728] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10408.082749] Lustre: 254642:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10408.154883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10414.176081] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10416.190260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10416.199652] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10416.302306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10425.962450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10426.322339] Lustre: lustre-OST0001: new disk, initializing [10426.329754] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10426.340395] Lustre: 255513:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10426.538560] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10434.495385] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10436.222295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10436.233322] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10436.376040] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10447.191866] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10456.852167] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10470.074375] Lustre: server umount lustre-MDT0000 complete [10471.396413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10471.407937] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10472.424270] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10474.322868] LustreError: 252999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668497 with bad export cookie 15824826448371257183 [10474.343669] LustreError: 252999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10474.353684] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10474.903079] Lustre: server umount lustre-MDT0001 complete [10489.902325] Lustre: server umount lustre-OST0000 complete [10493.732888] Lustre: 250514:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788668500/real 1788668500] req@ffff8da80ecfdf80 x1875554681827456/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788668516 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10493.766547] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10493.781900] Lustre: Skipped 2 previous similar messages [10494.376742] Lustre: server umount lustre-OST0001 complete [10510.899526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10521.869698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10531.577480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10542.276038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10564.794521] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10576.728634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10576.774727] Lustre: lustre-MDT0000: reset Object Index mappings [10577.613529] LustreError: 260523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10577.649223] LustreError: 260523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10577.722180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10582.451280] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10583.016988] LustreError: 260524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10588.139162] LustreError: 260523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10593.249885] LustreError: 260524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10593.284618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10593.311236] Lustre: lustre-MDT0001: reset Object Index mappings [10593.644649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10597.651636] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10600.618473] Lustre: 261664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10608.926953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10616.990500] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10618.622221] LustreError: 262017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10622.715438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10627.608132] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10627.857560] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10627.869476] Lustre: Skipped 1 previous similar message [10633.226503] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10635.723968] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10644.134785] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10648.787508] Lustre: 263538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10654.700393] Lustre: server umount lustre-MDT0000 complete [10655.207965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10655.222139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10655.245570] LustreError: 260524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10655.262232] LustreError: 260524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10658.461613] LustreError: 260505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668681 with bad export cookie 15824826448371263973 [10658.470136] LustreError: 260505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10658.481505] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10658.823536] Lustre: server umount lustre-MDT0001 complete [10674.145684] Lustre: 250512:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788668681/real 1788668681] req@ffff8da93770df80 x1875554681875840/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788668697 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10674.181287] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10679.199372] Lustre: 250511:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788668686/real 1788668686] req@ffff8da93770e680 x1875554681876480/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788668702 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10679.229684] Lustre: 250511:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10682.573985] Lustre: server umount lustre-OST0000 complete [10684.384086] Lustre: 250512:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788668691/real 1788668691] req@ffff8da9377d9f80 x1875554681876864/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788668707 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10684.430228] Lustre: 250512:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10687.427270] Lustre: server umount lustre-OST0001 complete [10709.983596] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:25:30 (1788668730) [10730.599440] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [10744.247974] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10790.279753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10790.640076] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10790.709618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10790.821725] Lustre: lustre-MDT0000: new disk, initializing [10790.940626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10790.971796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10796.461448] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10810.439718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10810.545429] Lustre: 267795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10810.599156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10810.604405] Lustre: Skipped 1 previous similar message [10810.690415] Lustre: lustre-MDT0001: new disk, initializing [10810.875494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10810.899994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10815.561552] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10826.864501] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10833.551530] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10841.871896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10842.193830] Lustre: lustre-OST0000: new disk, initializing [10842.198370] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10842.209542] Lustre: 269766:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10844.176819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10844.205342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10844.417081] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10849.579448] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10859.336324] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10863.655702] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10863.913927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10869.912468] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10870.187568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10875.361030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10875.370790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10875.381223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10875.392116] Lustre: Skipped 2 previous similar messages [10878.938726] Lustre: server umount lustre-OST0000 complete [10885.605667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10885.619299] Lustre: Skipped 2 previous similar messages [10889.440216] Lustre: server umount lustre-MDT0000 complete [10890.723255] LustreError: 267802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10890.732061] LustreError: 267802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10893.331723] LustreError: 267790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788668916 with bad export cookie 15824826448371266066 [10893.337124] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10893.351897] LustreError: 267790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10893.745640] Lustre: server umount lustre-MDT0001 complete [10917.092669] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [10928.120853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10928.567153] LustreError: 272701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10928.685741] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10928.697398] Lustre: Skipped 2 previous similar messages [10934.324800] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10945.155096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10952.131541] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10955.571661] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10962.385533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10969.782875] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10979.282823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10979.432491] Lustre: lustre-OST0001: new disk, initializing [10979.438627] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10979.445767] Lustre: 275010:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10981.053943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10981.062658] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10981.135463] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10986.343260] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [10997.548919] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11001.890222] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11007.538614] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11007.952929] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11022.309542] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11022.321425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11022.329487] Lustre: Skipped 5 previous similar messages [11022.360234] Lustre: Skipped 3 previous similar messages [11025.666757] Lustre: server umount lustre-MDT0000 complete [11027.428594] LustreError: 272701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11027.448370] LustreError: 272701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11029.199311] LustreError: 275009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669052 with bad export cookie 15824826448371266948 [11029.201950] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11029.212762] LustreError: 275009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11029.571853] Lustre: server umount lustre-MDT0001 complete [11044.153817] Lustre: server umount lustre-OST0000 complete [11058.794735] Lustre: server umount lustre-OST0001 complete [11083.800796] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11096.107625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11096.722248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11096.733662] Lustre: Skipped 3 previous similar messages [11101.764326] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11111.806665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11117.909928] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11121.442615] Lustre: 279499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11130.030873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11140.540504] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11142.008781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11154.371900] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11160.092370] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11161.865136] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11171.580282] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11176.451704] Lustre: 281378:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11183.363789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11183.414491] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11201.358191] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:33:41 (1788669221) [11219.436680] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11219.440626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11219.462748] Lustre: Skipped 3 previous similar messages [11219.478158] Lustre: Skipped 3 previous similar messages [11223.028936] Lustre: server umount lustre-MDT0000 complete [11224.550043] LustreError: 282129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11224.563992] LustreError: 282129:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [11227.591357] LustreError: 278340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669250 with bad export cookie 15824826448371273843 [11227.600517] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11227.609968] LustreError: 278340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11229.666856] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11229.679923] Lustre: Skipped 1 previous similar message [11234.192420] Lustre: server umount lustre-MDT0001 complete [11245.415987] Lustre: server umount lustre-OST0000 complete [11250.051396] Lustre: server umount lustre-OST0001 complete [11257.729988] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [11270.068810] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11326.568205] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11336.691473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11336.896777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11336.927568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11337.001673] Lustre: lustre-MDT0000: new disk, initializing [11337.119614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11342.015335] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11354.405928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11354.597356] Lustre: 285913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11354.999420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11355.003330] Lustre: Skipped 4 previous similar messages [11355.056853] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11360.637957] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11373.196483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11373.457621] Lustre: 287545:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11375.281069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11380.341389] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11391.437256] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11391.614558] Lustre: 288418:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11392.866918] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11392.876696] Lustre: Skipped 1 previous similar message [11392.965260] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11398.247380] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11409.603224] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11414.029695] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11414.036121] Lustre: Skipped 1 previous similar message [11431.499668] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:37:32 (1788669452) [11433.963465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11433.972027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11433.980267] Lustre: Skipped 5 previous similar messages [11433.993974] Lustre: Skipped 5 previous similar messages [11436.077151] Lustre: server umount lustre-MDT0000 complete [11441.027736] LustreError: 285906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669463 with bad export cookie 15824826448371281340 [11441.029417] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11441.038325] LustreError: 285906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11441.319992] Lustre: server umount lustre-MDT0001 complete [11456.509882] Lustre: server umount lustre-OST0000 complete [11460.576835] Lustre: 250511:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788669467/real 1788669467] req@ffff8da92f665180 x1875554682158336/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788669483 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11460.605144] Lustre: 250511:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11461.772628] Lustre: server umount lustre-OST0001 complete [11471.795626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11477.735423] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11482.598699] LustreError: 290666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11482.623585] LustreError: 290666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11488.835485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11494.743412] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11503.031861] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11509.835400] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11517.249112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11523.580059] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11531.930589] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11552.613263] loop1: detected capacity change from 0 to 204800000 [11552.912930] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11552.930063] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11552.946813] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11558.209737] loop1: detected capacity change from 0 to 204800000 [11563.601964] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11563.700073] LustreError: 290653:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs27855: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11563.709370] LustreError: 290653:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs27855 [11563.713286] LustreError: 290653:0:(mgs_handler.c:517:mgs_target_reg()) fs27855-MDT0001: Error checking target index: rc = -13 [11563.722151] LustreError: fs27855-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11563.728233] LustreError: 294385:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11563.733371] LustreError: 294385:0:(tgt_mount.c:2132:server_put_super()) no obd fs27855-MDT0001 [11563.737333] LustreError: 294385:0:(tgt_mount.c:116:server_deregister_mount()) fs27855-MDT0001 not registered [11563.780308] Lustre: server umount fs27855-MDT0001 complete [11563.783657] LustreError: 294385:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11575.283386] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11575.547455] Lustre: Setting parameter fs27855-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs27855-MDT0001 [11575.635413] Lustre: srv-fs27855-MDT0001: No data found on store. Initialize space. [11575.643833] Lustre: Skipped 4 previous similar messages [11575.821323] Lustre: fs27855-MDT0001: new disk, initializing [11575.823386] Lustre: Skipped 3 previous similar messages [11576.087359] Lustre: srv-fs27855-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11576.102237] Lustre: 294854:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27855-MDT0001-mdtlov: can't create new llog: rc = -115 [11576.124264] LustreError: 294854:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27855-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11582.095037] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11589.085977] Lustre: server umount fs27855-MDT0001 complete [11596.787331] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11603.626944] loop1: detected capacity change from 0 to 204800000 [11610.152658] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11610.577064] Lustre: 296421:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27855-MDT0001-mdtlov: can't create new llog: rc = -115 [11610.585893] LustreError: 296421:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27855-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11616.040617] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11617.266477] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11621.796921] Lustre: server umount fs27855-MDT0001 complete [11632.608204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11632.618637] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11632.625624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11632.649296] Lustre: Skipped 3 previous similar messages [11632.677106] Lustre: Skipped 2 previous similar messages [11638.227460] Lustre: server umount lustre-MDT0000 complete [11646.777821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11646.951906] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11651.944949] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11652.592656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11660.968077] loop1: detected capacity change from 0 to 204800000 [11667.540792] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11668.201454] Lustre: 298871:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs27855-MDT0001-mdtlov: can't create new llog: rc = -115 [11668.219296] LustreError: 298871:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs27855-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11673.258741] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11678.621535] Lustre: server umount fs27855-MDT0001 complete [11703.793933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11703.798062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11703.802779] Lustre: Skipped 5 previous similar messages [11706.869760] Lustre: server umount lustre-OST0000 complete [11714.032153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11716.784516] Lustre: server umount lustre-MDT0000 complete [11721.138356] LustreError: 290646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788669743 with bad export cookie 15824826448371284021 [11721.141491] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11721.152339] LustreError: 290646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11721.707890] Lustre: server umount lustre-MDT0001 complete [11733.770466] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [11737.136845] Key type lgssc unregistered [11737.544499] LNet: 301215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11737.552317] LNetError: 301215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11737.568321] LNet: Removed LNI 192.168.204.142@tcp [11738.706279] Key type .llcrypt unregistered [11738.710099] Key type ._llcrypt unregistered [11763.836541] Key type ._llcrypt registered [11763.838691] Key type .llcrypt registered [11763.942720] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [11765.004662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11765.074700] alg: No test for adler32 (adler32-zlib) [11766.161619] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [11766.480420] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [11768.161628] Key type lgssc registered [11769.486985] Lustre: Echo OBD driver; http://www.lustre.org/ [11777.727121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11777.756089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11779.128981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11783.899850] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11788.264462] LustreError: 302485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11788.295769] LustreError: 302485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11791.554621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11791.982977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11796.213992] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11803.422407] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11809.145759] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11817.638137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11818.106723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11823.225959] LustreError: 304312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11826.865553] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11828.722918] LustreError: 304315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11833.830324] LustreError: 304314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11835.063751] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11842.024783] LustreError: 304314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11842.046659] LustreError: 304314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11852.281178] LustreError: 304312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11852.314137] LustreError: 304312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11857.064972] loop1: detected capacity change from 0 to 204800000 [11857.308302] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11857.327970] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11857.349994] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11863.998300] loop1: detected capacity change from 0 to 204800000 [11870.849105] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11871.056322] Lustre: fs20420-OST0000: new disk, initializing [11871.068721] Lustre: srv-fs20420-OST0000: No data found on store. Initialize space. [11871.078065] Lustre: 306220:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11871.166898] Lustre: fs20420-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11872.754593] LustreError: 304987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11872.777615] LustreError: 304987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11877.930970] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11886.534211] Lustre: server umount fs20420-OST0000 complete [11899.418685] loop1: detected capacity change from 0 to 204800000 [11903.482576] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11905.359376] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11905.577808] Lustre: fs20420-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11908.599209] LustreError: 304987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11908.653611] LustreError: 304987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [11912.241070] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11918.852212] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11921.605598] loop2: detected capacity change from 0 to 204800000 [11921.916946] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11921.934356] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11921.964460] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11927.969697] loop2: detected capacity change from 0 to 204800000 [11934.026648] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11934.206363] LustreError: fs20420-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11934.216909] LustreError: fs20420-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11934.222698] LustreError: 309259:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11934.232524] LustreError: 309259:0:(tgt_mount.c:2132:server_put_super()) no obd fs20420-OST0001 [11934.243971] LustreError: 309259:0:(tgt_mount.c:116:server_deregister_mount()) fs20420-OST0001 not registered [11934.297221] Lustre: server umount fs20420-OST0001 complete [11934.300822] LustreError: 309259:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11946.175746] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11946.273193] Lustre: fs20420-OST0001: new disk, initializing [11946.276726] Lustre: srv-fs20420-OST0001: No data found on store. Initialize space. [11946.283988] Lustre: 309809:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11946.368981] Lustre: fs20420-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11952.545836] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [11961.998518] Lustre: server umount fs20420-OST0000 complete [11973.019201] Lustre: server umount fs20420-OST0001 complete [11975.146759] LustreError: 304987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11975.173390] LustreError: 304987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [11986.242539] loop1: detected capacity change from 0 to 204800000 [11991.888067] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11992.158855] Lustre: fs20420-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11998.388207] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12004.964200] loop2: detected capacity change from 0 to 204800000 [12010.597757] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12017.233650] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12027.327929] Lustre: server umount fs20420-OST0000 complete [12036.864991] Lustre: server umount fs20420-OST0001 complete [12048.864697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12048.871879] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12048.884574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12051.941227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12051.942017] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12054.800442] Lustre: server umount lustre-OST0000 complete [12062.179702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12062.192304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12062.206766] Lustre: Skipped 2 previous similar messages [12065.237509] Lustre: server umount lustre-MDT0000 complete [12068.947462] LustreError: 302467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670091 with bad export cookie 335955405142461095 [12068.949360] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12068.961320] LustreError: 302467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12069.425981] Lustre: server umount lustre-MDT0001 complete [12081.039692] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12083.678271] Key type lgssc unregistered [12083.994865] LNet: 315732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12084.004585] LNetError: 315732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12084.030415] LNet: Removed LNI 192.168.204.142@tcp [12084.804282] Key type .llcrypt unregistered [12084.810668] Key type ._llcrypt unregistered [12118.884432] Key type ._llcrypt registered [12118.886933] Key type .llcrypt registered [12119.049256] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [12136.107442] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [12137.088653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12137.249409] alg: No test for adler32 (adler32-zlib) [12138.421888] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [12138.723657] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [12140.471157] Key type lgssc registered [12141.572435] Lustre: Echo OBD driver; http://www.lustre.org/ [12178.851151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12178.928266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12180.318412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12180.390520] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12180.528700] Lustre: lustre-MDT0000: new disk, initializing [12180.726618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12180.768513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12186.515936] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12197.975340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12198.146532] Lustre: 319590:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12198.269448] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12198.275227] Lustre: Skipped 1 previous similar message [12198.356450] Lustre: lustre-MDT0001: new disk, initializing [12198.439100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12198.479103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12198.491701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12203.025622] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12213.093772] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12219.488246] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12226.383333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12226.688724] Lustre: lustre-OST0000: new disk, initializing [12226.692789] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12226.707366] Lustre: 321557:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12226.813972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12233.297979] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12234.341993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12234.352308] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12234.497777] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12244.074176] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12246.381824] LustreError: 317761:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0001_UUID failed: -22 [12246.429417] Lustre: Mounted lustre-client - version 2.17.58_2_g2c24a36 [12247.456640] Lustre: Unmounted lustre-client [12260.323688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12260.334626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12260.344095] LustreError: Skipped 1 previous similar message [12260.348433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12260.370629] Lustre: Skipped 1 previous similar message [12260.387430] Lustre: Skipped 1 previous similar message [12261.866650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12261.870201] Lustre: Skipped 1 previous similar message [12263.765555] Lustre: server umount lustre-OST0000 complete [12270.049241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12270.058498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12270.086627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12272.098277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12272.099431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12273.800330] Lustre: server umount lustre-MDT0000 complete [12277.296295] LustreError: 319580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670300 with bad export cookie 15284258501087625661 [12277.304299] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12277.311830] LustreError: 319580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12277.593209] Lustre: server umount lustre-MDT0001 complete [12288.178960] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12291.137984] Key type lgssc unregistered [12307.745507] LNet: 1 local NIs in recovery (showing 1): 192.168.204.142@tcp [12307.938130] LNet: 323716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12307.951402] LNetError: 323716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12375.531483] LNet: Removed LNI 192.168.204.142@tcp [12376.537218] Key type .llcrypt unregistered [12376.540897] Key type ._llcrypt unregistered [12396.389457] Key type ._llcrypt registered [12396.393558] Key type .llcrypt registered [12396.568748] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [12417.004727] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [12418.278592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12418.484146] alg: No test for adler32 (adler32-zlib) [12419.609074] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [12419.824978] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [12421.567272] Key type lgssc registered [12423.079532] Lustre: Echo OBD driver; http://www.lustre.org/ [12457.967424] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 00:54:38 (1788670478) [12465.992126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12466.035375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12467.339090] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12467.441646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12467.569643] Lustre: lustre-MDT0000: new disk, initializing [12467.671948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12467.709766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12472.302667] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12483.596947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12483.705963] Lustre: 326938:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12483.845218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12483.855874] Lustre: Skipped 1 previous similar message [12483.960985] Lustre: lustre-MDT0001: new disk, initializing [12484.130820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12484.207375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12484.225781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12490.604806] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12503.503572] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12511.057941] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12520.267646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12520.481215] Lustre: lustre-OST0000: new disk, initializing [12520.485232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12520.491789] Lustre: 328909:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12520.590670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12528.496917] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12528.730586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12528.743482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12528.888710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12539.186479] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12559.842693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12559.848595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12559.858277] LustreError: Skipped 1 previous similar message [12559.859293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12559.893853] Lustre: Skipped 1 previous similar message [12563.151872] Lustre: server umount lustre-OST0000 complete [12570.081261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12570.083020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12570.099394] Lustre: Skipped 1 previous similar message [12570.107312] Lustre: Skipped 2 previous similar messages [12575.201418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12575.216580] Lustre: Skipped 1 previous similar message [12579.302873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12579.313906] Lustre: Skipped 1 previous similar message [12581.343123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12581.654410] Lustre: server umount lustre-MDT0000 complete [12585.339379] LustreError: 327733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670608 with bad export cookie 16808799544521964423 [12585.350131] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12585.353612] LustreError: 327733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12585.713202] Lustre: server umount lustre-MDT0001 complete [12595.896410] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12598.694477] Key type lgssc unregistered [12598.990978] LNet: 330816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12599.001896] LNetError: 330816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12599.021818] LNet: Removed LNI 192.168.204.142@tcp [12600.167695] Key type .llcrypt unregistered [12600.172460] Key type ._llcrypt unregistered [12624.429628] Key type ._llcrypt registered [12624.434216] Key type .llcrypt registered [12624.530869] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [12625.351660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12625.403994] alg: No test for adler32 (adler32-zlib) [12626.433635] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [12626.786571] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [12628.511216] Key type lgssc registered [12629.621649] Lustre: Echo OBD driver; http://www.lustre.org/ [12637.592876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12637.632333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12639.260689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12644.965441] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12649.453634] LustreError: 332087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12649.489548] LustreError: 332087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12654.560826] LustreError: 332088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12656.636366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12657.134692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12662.491166] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12671.644275] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12678.510556] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12687.411406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12687.899053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12695.058779] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12704.175367] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12719.363344] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12720.977737] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12725.855246] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.42@tcp (stopping) [12728.803902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12728.824627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12728.864220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12730.349928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12730.354435] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12731.838140] Lustre: server umount lustre-MDT0000 complete [12735.462829] LustreError: 332087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12735.476665] LustreError: 332087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12736.274976] LustreError: 332070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670759 with bad export cookie 10732533853508994882 [12736.284317] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12736.285629] LustreError: 332070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12740.579874] LustreError: 332084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12740.581823] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12740.612329] LustreError: 332084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12740.656634] Lustre: Skipped 1 previous similar message [12740.663629] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12740.671826] Lustre: Skipped 3 previous similar messages [12742.768842] Lustre: server umount lustre-MDT0001 complete [12747.472938] Lustre: server umount lustre-OST0000 complete [12757.676425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12763.257647] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12766.545363] Lustre: 336270:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.142@tcp [12768.962313] Lustre: server umount MGS complete [12777.933042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12778.578319] LustreError: 336842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12778.606271] LustreError: 336842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12778.717620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12784.059192] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12787.261069] LustreError: 336838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12787.282066] LustreError: 336838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12792.992406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12793.454237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12798.947640] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12809.128078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12809.513384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12817.389447] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [12826.665369] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12828.558451] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12836.459920] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12838.932937] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12844.512226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12844.513503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12844.523733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12844.537756] Lustre: Skipped 1 previous similar message [12844.561772] Lustre: Skipped 3 previous similar messages [12846.483323] Lustre: server umount lustre-OST0000 complete [12854.754044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12854.761108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12854.771578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12857.309688] Lustre: server umount lustre-MDT0000 complete [12859.872730] LustreError: 336842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12859.897129] LustreError: 336842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12862.313512] LustreError: 336824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788670885 with bad export cookie 10732533853508995715 [12862.315575] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12862.330490] LustreError: 336824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12862.982586] Lustre: server umount lustre-MDT0001 complete [12877.147731] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [12880.336401] Key type lgssc unregistered [12880.634739] LNet: 340293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12880.648819] LNetError: 340293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12881.706280] LNet: Removed LNI 192.168.204.142@tcp [12882.638608] Key type .llcrypt unregistered [12882.647732] Key type ._llcrypt unregistered [12905.592746] Key type ._llcrypt registered [12905.596898] Key type .llcrypt registered [12905.834851] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [12926.746912] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [12928.320407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12928.411884] alg: No test for adler32 (adler32-zlib) [12929.653744] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [12929.965368] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [12931.687219] Key type lgssc registered [12933.232650] Lustre: Echo OBD driver; http://www.lustre.org/ [12981.484950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12981.521940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12982.829660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12982.872940] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12982.983215] Lustre: lustre-MDT0000: new disk, initializing [12983.099675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12983.142687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12989.298357] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13002.519367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13002.628766] Lustre: 343455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13002.672452] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13002.676820] Lustre: Skipped 1 previous similar message [13002.777250] Lustre: lustre-MDT0001: new disk, initializing [13002.901659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13002.952051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13002.968725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13008.497838] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13018.938876] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13025.213710] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13032.724103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13033.048172] Lustre: lustre-OST0000: new disk, initializing [13033.052346] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13033.064217] Lustre: 345425:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13033.184624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13040.137368] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13042.234572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13042.243091] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13042.336904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13050.510739] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13055.092068] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13055.309368] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13060.452557] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13060.712480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13063.137168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13063.153653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13063.167481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13063.177674] Lustre: Skipped 1 previous similar message [13068.258922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13068.266565] Lustre: Skipped 1 previous similar message [13069.001676] Lustre: server umount lustre-OST0000 complete [13073.378660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13073.380069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13073.413807] Lustre: Skipped 2 previous similar messages [13073.426734] Lustre: Skipped 1 previous similar message [13078.496909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13079.024414] Lustre: server umount lustre-MDT0000 complete [13082.996124] LustreError: 344259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671105 with bad export cookie 7795541777773505293 [13083.005867] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13083.014700] LustreError: 344259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13083.311733] Lustre: server umount lustre-MDT0001 complete [13094.595214] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 01:05:14 (1788671114) [13113.513323] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [13129.017876] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [13177.399505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13177.705725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13177.736054] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13177.851961] Lustre: lustre-MDT0000: new disk, initializing [13177.991616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13178.035662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13183.551866] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13196.215814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13196.400937] Lustre: 350208:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13196.472448] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13196.475919] Lustre: Skipped 1 previous similar message [13196.548639] Lustre: lustre-MDT0001: new disk, initializing [13196.628404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13196.667690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13196.678147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13201.220783] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13211.362656] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13218.431977] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13227.121402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13227.385754] Lustre: lustre-OST0000: new disk, initializing [13227.397397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13227.409680] Lustre: 352175:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13227.538366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13233.727922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13233.742273] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13233.845985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13234.759425] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [13243.707650] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13285.347512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13285.348248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13285.356663] LustreError: Skipped 2 previous similar messages [13285.358821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13285.377123] Lustre: Skipped 1 previous similar message [13285.394586] Lustre: Skipped 2 previous similar messages [13288.813470] Lustre: server umount lustre-OST0000 complete [13294.058944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13294.066084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13294.098221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13300.626211] Lustre: server umount lustre-MDT0000 complete [13304.993717] LustreError: 352174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788671327 with bad export cookie 7795541777773506175 [13305.001167] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13305.006179] LustreError: 352174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13305.369978] Lustre: server umount lustre-MDT0001 complete [13318.365132] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13321.211336] Key type lgssc unregistered [13321.568724] LNet: 354355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13321.578649] LNetError: 354355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13321.593572] LNet: Removed LNI 192.168.204.142@tcp [13322.683931] Key type .llcrypt unregistered [13322.687556] Key type ._llcrypt unregistered [13433.450629] Key type ._llcrypt registered [13433.452091] Key type .llcrypt registered [13433.570485] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13433.948434] Key type .llcrypt unregistered [13433.953893] Key type ._llcrypt unregistered [13522.202508] Key type ._llcrypt registered [13522.209378] Key type .llcrypt registered [13522.399627] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13522.818582] Key type .llcrypt unregistered [13522.826917] Key type ._llcrypt unregistered [13555.195433] Key type ._llcrypt registered [13555.198395] Key type .llcrypt registered [13555.312745] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13555.583638] Key type .llcrypt unregistered [13555.588282] Key type ._llcrypt unregistered [13586.877174] Key type ._llcrypt registered [13586.880630] Key type .llcrypt registered [13586.987858] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13587.356483] Key type .llcrypt unregistered [13587.361777] Key type ._llcrypt unregistered [13671.504709] Key type ._llcrypt registered [13671.507442] Key type .llcrypt registered [13671.616409] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13671.990284] Key type .llcrypt unregistered [13671.999889] Key type ._llcrypt unregistered [13705.698082] Key type ._llcrypt registered [13705.701265] Key type .llcrypt registered [13705.861254] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13706.142299] Key type .llcrypt unregistered [13706.146797] Key type ._llcrypt unregistered [13739.722824] Key type ._llcrypt registered [13739.735166] Key type .llcrypt registered [13739.906714] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13740.190492] Key type .llcrypt unregistered [13740.192383] Key type ._llcrypt unregistered [13775.403230] Key type ._llcrypt registered [13775.408491] Key type .llcrypt registered [13775.549985] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13775.842660] Key type .llcrypt unregistered [13775.847731] Key type ._llcrypt unregistered [13806.422874] Key type ._llcrypt registered [13806.425492] Key type .llcrypt registered [13806.531916] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13806.865436] Key type .llcrypt unregistered [13806.870863] Key type ._llcrypt unregistered [13840.139398] Key type ._llcrypt registered [13840.142746] Key type .llcrypt registered [13840.250508] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13840.603294] Key type .llcrypt unregistered [13840.608207] Key type ._llcrypt unregistered [13906.581303] Key type ._llcrypt registered [13906.587572] Key type .llcrypt registered [13906.711533] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13906.990674] Key type .llcrypt unregistered [13906.995668] Key type ._llcrypt unregistered [13937.835567] Key type ._llcrypt registered [13937.839032] Key type .llcrypt registered [13938.004861] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13938.352502] Key type .llcrypt unregistered [13938.354795] Key type ._llcrypt unregistered [13973.763478] Key type ._llcrypt registered [13973.765073] Key type .llcrypt registered [13973.916681] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [13974.315658] Key type .llcrypt unregistered [13974.321230] Key type ._llcrypt unregistered [14034.735459] Key type ._llcrypt registered [14034.739161] Key type .llcrypt registered [14034.894143] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [14054.118142] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [14055.232669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14055.306673] alg: No test for adler32 (adler32-zlib) [14056.322180] Lustre: Lustre: Build Version: 2.17.58_2_g2c24a36 [14056.687225] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [14058.311183] Key type lgssc registered [14059.501255] Lustre: Echo OBD driver; http://www.lustre.org/ [14097.781857] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14097.812603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14099.091634] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14099.138495] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14099.246195] Lustre: tst250fs-MDT0000: new disk, initializing [14099.379300] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14099.414498] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14104.925991] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [14116.798851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14116.886712] Lustre: 374663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14117.023675] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14117.036093] Lustre: Skipped 1 previous similar message [14117.151973] Lustre: tst250fs-MDT0001: new disk, initializing [14117.300027] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14117.393631] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14117.404618] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14123.104251] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [14129.087662] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14137.520928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14137.797252] Lustre: tst250fs-OST0000: new disk, initializing [14137.801174] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14137.808252] Lustre: 376293:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14137.908620] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14143.582405] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14143.592962] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14143.694670] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14144.081550] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [14155.930860] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14156.214910] Lustre: tst250fs-OST0001: new disk, initializing [14156.229861] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14156.281432] Lustre: 377165:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14156.425966] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14162.932191] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [14163.041147] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14163.053661] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14163.199617] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14169.175910] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14171.270789] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14171.279633] Lustre: Skipped 1 previous similar message [14194.151723] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14194.158081] Lustre: Skipped 1 previous similar message [14194.158896] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14194.162168] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14194.182022] Lustre: Skipped 3 previous similar messages [14197.595635] Lustre: server umount tst250fs-MDT0000 complete [14202.030966] LustreError: 377164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788672224 with bad export cookie 17442784257119867199 [14202.034243] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14202.054429] LustreError: 377164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14202.456301] Lustre: server umount tst250fs-MDT0001 complete [14217.228614] Lustre: server umount tst250fs-OST0000 complete [14220.258439] Lustre: 372882:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788672227/real 1788672227] req@ffff8da8101fca80 x1875558613983360/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788672243 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14220.292106] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14220.308315] Lustre: Skipped 2 previous similar messages [14221.688895] Lustre: server umount tst250fs-OST0001 complete [14240.570863] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [14248.805590] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [14283.540935] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 01:25:03 (1788672303) [14285.721193] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14305.320654] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [14315.626228] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [14349.829320] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14084 sec ========= 01:26:10 (1788672370) [14351.733962] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:26:12 (1788672372) === [14355.841892] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:26:16 (1788672376) === [14399.626894] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [14402.441118] Key type lgssc unregistered [14402.758430] LNet: 386309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14402.762517] LNetError: 386309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14403.818046] LNet: Removed LNI 192.168.204.142@tcp [14404.827167] Key type .llcrypt unregistered [14404.829167] Key type ._llcrypt unregistered