[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 470243419 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2592MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003152] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008460] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011137] LSM: Security Framework initializing [ 0.012057] Yama: becoming mindful. [ 0.013041] SELinux: Initializing. [ 0.014085] *** VALIDATE selinux *** [ 0.022188] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027413] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028177] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031027] *** VALIDATE tmpfs *** [ 0.032417] *** VALIDATE proc *** [ 0.033239] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.036064] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038029] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040027] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.043781] debug: unmapping init [mem 0xffffffffa9c59000-0xffffffffa9c60fff] [ 0.046118] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047720] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048037] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050015] ... generic registers: 4 [ 0.051018] ... value mask: 0000ffffffffffff [ 0.052018] ... max period: 00007fffffffffff [ 0.053011] ... fixed-purpose events: 3 [ 0.053981] ... event mask: 000000070000000f [ 0.054274] rcu: Hierarchical SRCU implementation. [ 0.056407] smp: Bringing up secondary CPUs ... [ 0.057596] x86: Booting SMP configuration: [ 0.058024] .... node #0, CPUs: #1 #2 #3 [ 0.061465] smp: Brought up 1 node, 4 CPUs [ 0.063010] smpboot: Max logical packages: 1 [ 0.064016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141478] node 0 deferred pages initialised in 74ms [ 0.143112] devtmpfs: initialized [ 0.144230] x86/mm: Memory block size: 128MB [ 0.147278] gcov: version magic: 0x41383552 [ 0.149390] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.153089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154289] pinctrl core: initialized pinctrl subsystem [ 0.155191] [ 0.155872] ************************************************************* [ 0.156013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157014] ** ** [ 0.158016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.159015] ** ** [ 0.160015] ** This means that this kernel is built to expose internal ** [ 0.161015] ** IOMMU data structures, which may compromise security on ** [ 0.162014] ** your system. ** [ 0.163014] ** ** [ 0.164014] ** If you see this message and you are not debugging the ** [ 0.165014] ** kernel, report this immediately to your vendor! ** [ 0.166014] ** ** [ 0.167015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168016] ************************************************************* [ 0.169677] NET: Registered protocol family 16 [ 0.170000] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172044] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.174040] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177086] cpuidle: using governor menu [ 0.178692] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.181486] PCI: Using configuration type 1 for base access [ 0.183102] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.191115] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.193038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.197063] cryptd: max_cpu_qlen set to 1000 [ 0.199260] ACPI: Added _OSI(Module Device) [ 0.201015] ACPI: Added _OSI(Processor Device) [ 0.203014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.204013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.208978] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.215035] ACPI: Interpreter enabled [ 0.215957] ACPI: PM: (supports S0 S3 S4 S5) [ 0.217020] ACPI: Using IOAPIC for interrupt routing [ 0.218000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.220387] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.228748] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.231034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.232013] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.234060] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.237998] acpiphp: Slot [2] registered [ 0.238904] acpiphp: Slot [5] registered [ 0.240084] acpiphp: Slot [6] registered [ 0.241071] acpiphp: Slot [7] registered [ 0.242111] acpiphp: Slot [8] registered [ 0.243087] acpiphp: Slot [9] registered [ 0.244082] acpiphp: Slot [10] registered [ 0.245092] acpiphp: Slot [3] registered [ 0.247066] acpiphp: Slot [4] registered [ 0.248114] acpiphp: Slot [11] registered [ 0.250208] acpiphp: Slot [12] registered [ 0.251228] acpiphp: Slot [13] registered [ 0.253104] acpiphp: Slot [14] registered [ 0.255108] acpiphp: Slot [15] registered [ 0.256205] acpiphp: Slot [16] registered [ 0.258132] acpiphp: Slot [17] registered [ 0.259153] acpiphp: Slot [18] registered [ 0.261183] acpiphp: Slot [19] registered [ 0.263206] acpiphp: Slot [20] registered [ 0.264135] acpiphp: Slot [21] registered [ 0.266142] acpiphp: Slot [22] registered [ 0.268127] acpiphp: Slot [23] registered [ 0.269114] acpiphp: Slot [24] registered [ 0.271206] acpiphp: Slot [25] registered [ 0.272105] acpiphp: Slot [26] registered [ 0.274118] acpiphp: Slot [27] registered [ 0.275136] acpiphp: Slot [28] registered [ 0.277117] acpiphp: Slot [29] registered [ 0.279152] acpiphp: Slot [30] registered [ 0.280139] acpiphp: Slot [31] registered [ 0.282101] PCI host bridge to bus 0000:00 [ 0.283026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.286022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.288026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.291025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.294030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.296025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.298162] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.301212] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.305227] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.314068] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.320028] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.322018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.325019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.327018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.330561] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.332748] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.336047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.338067] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.344018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.357015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.361959] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.366703] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.383017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.392018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.418026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.434077] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.444018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.460018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.492017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.503333] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.512022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.521016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.542016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.555239] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.573020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.588028] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.628025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.654056] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.666017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.680020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.704017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.713760] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.720015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.726017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.737018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.745926] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.748360] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.750301] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.753384] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.756268] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.761020] iommu: Default domain type: Passthrough [ 0.762393] SCSI subsystem initialized [ 0.763130] ACPI: bus type USB registered [ 0.765121] usbcore: registered new interface driver usbfs [ 0.767088] usbcore: registered new interface driver hub [ 0.769082] usbcore: registered new device driver usb [ 0.770191] pps_core: LinuxPPS API ver. 1 registered [ 0.772013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.775064] PTP clock support registered [ 0.777111] EDAC MC: Ver: 3.0.0 [ 0.779502] PCI: Using ACPI for IRQ routing [ 0.781873] NetLabel: Initializing [ 0.783011] NetLabel: domain hash size = 128 [ 0.785012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.787087] NetLabel: unlabeled traffic allowed by default [ 0.789122] vgaarb: loaded [ 0.791243] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.792018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.799650] clocksource: Switched to clocksource kvm-clock [ 0.904672] VFS: Disk quotas dquot_6.6.0 [ 0.906165] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.909344] *** VALIDATE ramfs *** [ 0.910671] *** VALIDATE hugetlbfs *** [ 0.912241] pnp: PnP ACPI init [ 0.914659] pnp: PnP ACPI: found 6 devices [ 0.929895] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.933095] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.935300] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.937464] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.939909] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.942025] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.944701] NET: Registered protocol family 2 [ 0.947114] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.951737] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.954949] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.959495] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.962978] TCP: Hash tables configured (established 65536 bind 65536) [ 0.965914] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.968181] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.970451] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.973157] NET: Registered protocol family 1 [ 0.975507] RPC: Registered named UNIX socket transport module. [ 0.977853] RPC: Registered udp transport module. [ 0.979516] RPC: Registered tcp transport module. [ 0.981270] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.983650] NET: Registered protocol family 44 [ 0.985347] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.987529] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.989744] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.992156] PCI: CLS 0 bytes, default 64 [ 0.993866] Unpacking initramfs... [ 2.422276] debug: unmapping init [mem 0xffff9038bcc54000-0xffff9038bffbffff] [ 2.429127] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.431193] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.434416] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.003734] Initialise system trusted keyrings [ 3.005468] Key type blacklist registered [ 3.007406] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.017211] zbud: loaded [ 3.021489] *** VALIDATE nfs *** [ 3.023274] *** VALIDATE nfs4 *** [ 3.024949] pstore: using deflate compression [ 3.028798] Platform Keyring initialized [ 3.147031] NET: Registered protocol family 38 [ 3.148927] Key type asymmetric registered [ 3.150682] Asymmetric key parser 'x509' registered [ 3.152614] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.155800] io scheduler mq-deadline registered [ 3.157489] io scheduler kyber registered [ 3.159274] io scheduler bfq registered [ 3.161386] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.164594] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.167658] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.170570] ACPI: Power Button [PWRF] [ 3.175991] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.183267] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.205470] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.230116] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.254230] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.283517] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.312928] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.318369] Non-volatile memory driver v1.3 [ 3.319979] Linux agpgart interface v0.103 [ 3.358208] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.361390] vda: detected capacity change from 0 to 74702848 [ 3.376196] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.379240] vdb: detected capacity change from 0 to 1073741824 [ 3.409272] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.412526] vdc: detected capacity change from 0 to 2621440000 [ 3.432065] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.436449] vdd: detected capacity change from 0 to 2621440000 [ 3.452720] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.455516] vde: detected capacity change from 0 to 4294967296 [ 3.475533] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.479983] vdf: detected capacity change from 0 to 4294967296 [ 3.489637] libphy: Fixed MDIO Bus: probed [ 3.502703] usbcore: registered new interface driver usbserial_generic [ 3.505560] usbserial: USB Serial support registered for generic [ 3.508292] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.512878] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.515080] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.517676] mousedev: PS/2 mouse device common for all mice [ 3.521416] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.524973] rtc_cmos 00:05: RTC can wake from S4 [ 3.527962] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.529718] rtc_cmos 00:05: registered as rtc0 [ 3.535472] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.536083] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.541613] intel_pstate: CPU model not supported [ 3.544323] hid: raw HID events driver (C) Jiri Kosina [ 3.546268] usbcore: registered new interface driver usbhid [ 3.548123] usbhid: USB HID core driver [ 3.549672] drop_monitor: Initializing network drop monitor service [ 3.551813] Initializing XFRM netlink socket [ 3.553560] NET: Registered protocol family 10 [ 3.556227] Segment Routing with IPv6 [ 3.557626] NET: Registered protocol family 17 [ 3.559236] mpls_gso: MPLS GSO support [ 3.564396] RAS: Correctable Errors collector initialized. [ 3.566319] AVX version of gcm_enc/dec engaged. [ 3.568048] AES CTR mode by8 optimization enabled [ 3.668105] sched_clock: Marking stable (3668075400, 0)->(4589130597, -921055197) [ 3.671696] registered taskstats version 1 [ 3.673660] Loading compiled-in X.509 certificates [ 3.675721] zswap: loaded using pool lzo/zbud [ 3.699867] Key type big_key registered [ 3.712899] Key type encrypted registered [ 3.714713] ima: No TPM chip found, activating TPM-bypass! [ 3.716903] ima: Allocated hash algorithm: sha1 [ 3.719248] ima: No architecture policies found [ 3.721099] evm: Initialising EVM extended attributes: [ 3.722987] evm: security.selinux [ 3.724359] evm: security.ima [ 3.725261] evm: security.capability [ 3.726350] evm: HMAC attrs: 0x1 [ 3.728372] rtc_cmos 00:05: setting system clock to 2026-08-09 19:50:44 UTC (1786305044) [ 3.735678] debug: unmapping init [mem 0xffffffffaac03000-0xffffffffaadfffff] [ 3.739095] debug: unmapping init [mem 0xffffffffa9982000-0xffffffffa9c58fff] [ 3.744124] Write protecting the kernel read-only data: 28672k [ 3.747268] debug: unmapping init [mem 0xffffffffa8003000-0xffffffffa81fffff] [ 3.750134] debug: unmapping init [mem 0xffffffffa8914000-0xffffffffa89fffff] [ 3.803747] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.813991] systemd[1]: Detected virtualization kvm. [ 3.815606] systemd[1]: Detected architecture x86-64. [ 3.817369] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.847485] systemd[1]: No hostname configured. [ 3.849220] systemd[1]: Set hostname to . [ 3.851415] random: systemd: uninitialized urandom read (16 bytes read) [ 3.854281] systemd[1]: Initializing machine ID from random generator. [ 3.906895] random: ln: uninitialized urandom read (6 bytes read) [ 4.000308] random: systemd: uninitialized urandom read (16 bytes read) [ 4.003181] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.008633] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.017974] systemd[1]: Starting Create Volatile Files and Directories... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.673043] device-mapper: uevent: version 1.0.3 [ 4.676164] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.447474] virtio_net virtio0 ens2: renamed from eth0 [ 5.472120] random: fast init done [ 5.536172] scsi host0: ata_piix [ 5.571666] scsi host1: ata_piix [ 5.573466] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.576081] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.945882] dracut-initqueue[594]: RTNETLINK answers: File exists [ 10.231675] random: crng init done [ 10.233116] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.717915] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.962305] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.236421] SELinux: Disabled at runtime. [ 12.290515] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.300663] systemd[1]: Detected virtualization kvm. [ 12.302693] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.792485] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.797390] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.802493] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.806314] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.808526] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.814955] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.818771] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK [[ 12.897988] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.272836] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ OK ] Started udev Coldplug all Devices. [ 13.609959] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.672699] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.817145] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.826872] EDAC sbridge: Ver: 1.1.2 [ 15.513425] Key type dns_resolver registered [ 15.837955] NFS: Registering the id_resolver key type [ 15.840108] Key type id_resolver registered [ 15.841853] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg202-server login: [ 80.104562] hrtimer: interrupt took 2661484 ns [ 98.940630] libcfs: loading out-of-tree module taints kernel. [ 99.057924] Key type ._llcrypt registered [ 99.063789] Key type .llcrypt registered [ 99.263682] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 124.158566] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 126.544676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 126.569844] alg: No test for adler32 (adler32-zlib) [ 128.239539] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 129.437879] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 131.312387] Key type lgssc registered [ 134.130968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 156.308245] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 215.415841] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 234.451377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 234.568145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 236.336259] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 236.433839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 236.661119] Lustre: lustre-MDT0000: new disk, initializing [ 236.896255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 236.960703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 245.061139] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 264.534464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 264.723890] Lustre: 6532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 264.796078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 264.805966] Lustre: Skipped 1 previous similar message [ 264.992674] Lustre: lustre-MDT0001: new disk, initializing [ 265.089917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 265.126410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 265.133807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 270.418987] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.302748] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 288.710485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 289.059970] Lustre: lustre-OST0000: new disk, initializing [ 289.065368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 289.074065] Lustre: 8471:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 289.168983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 295.454950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 295.466580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 295.595795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 296.841910] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 314.983170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 315.176880] Lustre: lustre-OST0001: new disk, initializing [ 315.181139] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 315.187539] Lustre: 9544:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 315.251939] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 323.740214] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 324.737684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 324.754179] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 324.842422] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 338.768875] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 347.537259] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 355.265487] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing check_logdir /tmp/testlogs/ [ 361.892032] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing yml_node [ 367.833000] Lustre: DEBUG MARKER: Client: 2.17.56.59 [ 371.207568] Lustre: DEBUG MARKER: MDS: 2.17.56.59 [ 374.396899] Lustre: DEBUG MARKER: OSS: 2.17.56.59 [ 376.371944] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 9 15:56:55 EDT 2026 [ 398.427936] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 400.399800] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 418.784355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 418.791832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 418.815082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 422.379338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 422.380851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 422.391971] Lustre: Skipped 1 previous similar message [ 422.408885] Lustre: Skipped 3 previous similar messages [ 424.335924] Lustre: server umount lustre-MDT0000 complete [ 428.876495] LustreError: 6522:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786305469 with bad export cookie 11092583710081447864 [ 428.877065] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 428.893331] LustreError: 6522:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 432.609269] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 432.609499] LustreError: 6537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 432.611721] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 432.623663] Lustre: Skipped 2 previous similar messages [ 432.647618] LustreError: 6537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 433.634444] LustreError: 6538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 433.644297] LustreError: 6538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 438.752518] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 438.755138] LustreError: 6538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 438.769279] Lustre: Skipped 4 previous similar messages [ 438.788950] LustreError: 6538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 443.360033] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 443.799443] Lustre: server umount lustre-MDT0001 complete [ 448.204386] Lustre: server umount lustre-OST0000 complete [ 453.455512] Lustre: server umount lustre-OST0001 complete [ 462.059623] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 474.834817] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 515.490599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.794081] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 515.823674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 515.938461] Lustre: lustre-MDT0000: new disk, initializing [ 516.083248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 516.141929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 521.596812] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 532.469355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 532.604914] Lustre: 15105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 532.688334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 532.692735] Lustre: Skipped 1 previous similar message [ 532.797486] Lustre: lustre-MDT0001: new disk, initializing [ 532.930423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 532.970828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 532.990782] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 538.965461] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 553.180420] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 562.852199] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 571.678205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 572.140571] Lustre: lustre-OST0000: new disk, initializing [ 572.153525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 572.164681] Lustre: 17075:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 572.336814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 574.007013] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 574.030378] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 574.270072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 581.353302] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 593.733034] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 599.242555] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 599.375879] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 604.978647] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 605.256161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 610.272501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 610.285767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.310578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 610.319481] Lustre: Skipped 1 previous similar message [ 613.670503] Lustre: server umount lustre-OST0000 complete [ 620.000982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 620.004405] LustreError: Skipped 1 previous similar message [ 620.008450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.026464] Lustre: Skipped 1 previous similar message [ 620.031565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 632.287178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 632.564751] Lustre: server umount lustre-MDT0000 complete [ 635.875738] LustreError: 15115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.899472] LustreError: 15115:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 637.100318] LustreError: 15097:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786305677 with bad export cookie 11092583710081449397 [ 637.109039] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 637.114909] LustreError: 15097:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 1 previous similar message [ 637.507874] Lustre: server umount lustre-MDT0001 complete [ 644.286781] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 16:01:23 (1786305683) [ 668.188485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 668.676639] LustreError: 19964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 668.790902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 674.804596] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 678.371609] LustreError: 19965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 678.392880] LustreError: 19965:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 686.641704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 693.198791] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 703.362543] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 710.013297] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 712.210561] Lustre: Failing over lustre-MDT0000 [ 712.510255] Lustre: server umount lustre-MDT0000 complete [ 712.675718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 712.693471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 712.705662] Lustre: Skipped 1 previous similar message [ 712.711444] LustreError: 19965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.731760] LustreError: 19965:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 733.309261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 733.395967] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 733.734258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 733.747039] Lustre: Skipped 1 previous similar message [ 733.815319] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 738.801649] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 738.808964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 738.836603] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 739.101419] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 750.530798] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 756.714190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.717943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 756.733570] Lustre: Skipped 2 previous similar messages [ 756.737036] Lustre: Skipped 6 previous similar messages [ 761.045481] Lustre: server umount lustre-MDT0000 complete [ 761.828593] LustreError: 19959:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 761.851915] LustreError: 19959:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 765.364164] LustreError: 19945:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786305806 with bad export cookie 11092583710081450811 [ 765.370170] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 765.373715] LustreError: 19945:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 765.778412] Lustre: server umount lustre-MDT0001 complete [ 777.105504] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 782.631556] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 792.585291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 799.662636] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 810.118560] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 816.918903] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 819.969102] Lustre: Failing over lustre-MDT0000 [ 820.358898] Lustre: server umount lustre-MDT0000 complete [ 822.763713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 822.783056] Lustre: Skipped 1 previous similar message [ 827.873126] LustreError: 23306:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 827.908879] LustreError: 23306:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 838.944570] Lustre: 3659:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786305863/real 1786305863] req@ffff90393f4e8a80 x1873076730145152/t0(0) o400->MGC192.168.202.102@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786305879 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 839.031336] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 842.096041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 848.353583] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x99f0c61d00b086e5 [ 848.370535] Lustre: MGC192.168.202.102@tcp: Connection restored to 0@lo (at 0@lo) [ 848.378100] Lustre: Skipped 1 previous similar message [ 848.845827] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 853.987378] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 853.994289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 854.027256] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 854.405148] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 864.746551] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 869.355970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 869.366208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 869.368695] Lustre: Skipped 1 previous similar message [ 875.606474] Lustre: server umount lustre-MDT0000 complete [ 879.285241] LustreError: 23292:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786305920 with bad export cookie 11092583710081451749 [ 879.295989] LustreError: 23292:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 879.297891] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 879.726100] Lustre: server umount lustre-MDT0001 complete [ 888.999821] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 16:05:28 (1786305928) [ 905.466278] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 918.455073] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 971.240189] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 983.771552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 984.210460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 984.249462] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 984.399667] Lustre: lustre-MDT0000: new disk, initializing [ 984.564771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 984.589118] Lustre: Skipped 3 previous similar messages [ 984.616403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 989.632714] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1000.955366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.088719] Lustre: 29707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1001.663329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1006.845588] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1019.025161] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1027.558797] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1037.059210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1037.914436] Lustre: 31680:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1039.420573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1039.503845] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1039.708302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1039.775745] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 1047.993843] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1059.561310] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1066.710969] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1101.292289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1101.294920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1101.322280] Lustre: Skipped 2 previous similar messages [ 1101.324959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1101.362803] Lustre: Skipped 3 previous similar messages [ 1104.846427] Lustre: server umount lustre-OST0000 complete [ 1115.541332] Lustre: server umount lustre-MDT0000 complete [ 1116.649644] LustreError: 29718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.675636] LustreError: 29718:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 1119.422133] LustreError: 31678:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786306160 with bad export cookie 11092583710081452162 [ 1119.436806] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1120.244770] Lustre: server umount lustre-MDT0001 complete [ 1135.065462] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1139.305187] Key type lgssc unregistered [ 1139.783130] LNet: 33667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1139.799469] LNetError: 33667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1139.828835] LNet: Removed LNI 192.168.202.102@tcp [ 1141.346375] Key type .llcrypt unregistered [ 1141.353900] Key type ._llcrypt unregistered [ 1176.042370] Key type ._llcrypt registered [ 1176.044029] Key type .llcrypt registered [ 1176.182065] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1177.420450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1177.563082] alg: No test for adler32 (adler32-zlib) [ 1178.731122] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 1179.052688] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1180.919942] Key type lgssc registered [ 1182.301966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1191.040836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1191.074876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1192.636935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1197.267854] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1203.175979] LustreError: 35082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1203.189705] LustreError: 35082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1206.366455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1206.902271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1211.812666] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1221.114373] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1227.832559] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1235.663209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1236.173072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1238.311201] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1243.422741] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1251.597696] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1253.204502] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1269.751946] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 16:11:49 (1786306309) [ 1271.948863] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1273.842475] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1282.041120] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 16:12:01 (1786306321) [ 1294.640453] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 16:12:13 (1786306333) [ 1295.822924] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1307.733138] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1307.753094] Lustre: Skipped 2 previous similar messages [ 1310.082711] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 16:12:29 (1786306349) [ 1312.502660] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1312.510197] Lustre: Skipped 2 previous similar messages [ 1322.260568] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1322.270429] Lustre: Skipped 4 previous similar messages [ 1339.979073] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 16:12:59 (1786306379) [ 1343.365926] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1348.334150] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1360.009883] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 16:13:18 (1786306398) [ 1362.430584] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1362.442226] Lustre: Skipped 6 previous similar messages [ 1378.829856] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 16:13:37 (1786306417) [ 1397.219341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1397.229870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.239109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1400.289918] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1400.293212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1400.323145] Lustre: Skipped 1 previous similar message [ 1402.605599] Lustre: server umount lustre-OST0000 complete [ 1406.433634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1406.436938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1406.444212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1410.528838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1410.532234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1412.636675] Lustre: server umount lustre-MDT0000 complete [ 1415.659560] LustreError: 35082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1415.679839] LustreError: 35082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1416.234289] LustreError: 35063:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786306457 with bad export cookie 15999217667698733752 [ 1416.244760] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1416.526563] Lustre: server umount lustre-MDT0001 complete [ 1430.432268] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 1434.175895] Key type lgssc unregistered [ 1434.515518] LNet: 42949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1434.522086] LNetError: 42949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1434.552098] LNet: Removed LNI 192.168.202.102@tcp [ 1435.954661] Key type .llcrypt unregistered [ 1435.957330] Key type ._llcrypt unregistered [ 1460.726897] Key type ._llcrypt registered [ 1460.729735] Key type .llcrypt registered [ 1460.916541] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 1482.710908] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 1484.829344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1484.974958] alg: No test for adler32 (adler32-zlib) [ 1486.149574] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 1486.547314] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 1488.352169] Key type lgssc registered [ 1490.034777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1537.037339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1537.082592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1538.396938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1538.468970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1538.615714] Lustre: lustre-MDT0000: new disk, initializing [ 1538.770536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1538.806369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1544.523921] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1556.614186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1556.836288] Lustre: 46118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1556.941804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1556.957073] Lustre: Skipped 1 previous similar message [ 1557.091187] Lustre: lustre-MDT0001: new disk, initializing [ 1557.205869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1557.253904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1557.275404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1562.927921] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1575.232418] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1582.388301] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1590.413462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1590.825213] Lustre: lustre-OST0000: new disk, initializing [ 1590.843888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1590.859789] Lustre: 48086:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1590.972315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1598.130950] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1601.154787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1601.170672] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1601.277816] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1609.497820] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1614.983568] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1615.215696] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1620.932327] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1621.106662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1627.107748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1627.109536] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1627.123223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1627.135283] Lustre: Skipped 1 previous similar message [ 1629.538382] Lustre: server umount lustre-OST0000 complete [ 1634.272085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1634.278653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1634.308907] Lustre: Skipped 1 previous similar message [ 1634.321803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1637.346647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1637.349109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.749816] Lustre: server umount lustre-MDT0000 complete [ 1643.640315] LustreError: 46110:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786306684 with bad export cookie 14916868678367807926 [ 1643.653469] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1643.657487] LustreError: 46110:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1643.969699] Lustre: server umount lustre-MDT0001 complete [ 1655.432527] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 16:18:14 (1786306694) [ 1664.109518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1664.757222] LustreError: 50547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1664.786574] LustreError: 50547:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1664.874413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1670.116417] LustreError: 50548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1670.585439] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1675.234707] LustreError: 50547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1680.366039] LustreError: 50548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1681.956309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1682.458220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1688.253652] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1699.416919] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1708.068416] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1717.273455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1717.805250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1725.444339] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 1735.629136] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1740.863770] LustreError: 53079:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1741.905380] Lustre: Modifying parameter general.timeout=1 in log params [ 1743.448252] Lustre: Modifying parameter general.timeout=2 in log params [ 1745.853234] Lustre: Modifying parameter general.timeout=4 in log params [ 1745.857253] Lustre: Skipped 1 previous similar message [ 1750.226214] Lustre: Modifying parameter general.timeout=8 in log params [ 1750.234713] Lustre: Skipped 3 previous similar messages [ 1759.515457] Lustre: Modifying parameter general.timeout=15 in log params [ 1759.523793] Lustre: Skipped 6 previous similar messages [ 1775.959706] Lustre: Modifying parameter general.timeout=29 in log params [ 1775.961731] Lustre: Skipped 13 previous similar messages [ 1808.249990] Lustre: Modifying parameter general.timeout=61 in log params [ 1808.254618] Lustre: Skipped 31 previous similar messages [ 1872.734131] Lustre: Modifying parameter general.timeout=121 in log params [ 1872.741128] Lustre: Skipped 59 previous similar messages [ 1892.145648] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 16:22:11 (1786306931) [ 1895.184209] Lustre: 60037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1897.458457] Lustre: 60136:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1899.545891] Lustre: 60250:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1903.565227] Lustre: 60447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1903.575570] Lustre: 60447:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1932.850103] Lustre: 61920:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1932.858517] Lustre: 61920:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1932.877367] LustreError: 61920:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1937.182353] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 16:22:56 (1786306976) [ 1953.766276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1953.774028] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.786745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1953.789645] Lustre: Skipped 1 previous similar message [ 1957.494794] Lustre: server umount lustre-MDT0000 complete [ 1959.919942] LustreError: 51619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.942856] LustreError: 51619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1961.604967] LustreError: 50528:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786307002 with bad export cookie 14916868678367808808 [ 1961.609772] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1961.620075] LustreError: 50528:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1962.463784] Lustre: server umount lustre-MDT0001 complete [ 1977.700994] Lustre: server umount lustre-OST0000 complete [ 2009.610812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2010.228099] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2010.324654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2010.342469] Lustre: Skipped 31 previous similar messages [ 2010.849973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2015.381705] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2024.004440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.176796] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2024.244720] Lustre: 64455:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2029.429252] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2039.268879] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2046.608773] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2055.333695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2055.548083] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2055.996222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2056.006573] Lustre: Skipped 1 previous similar message [ 2063.418723] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2073.807923] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2076.327920] Lustre: 67023:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2082.279494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2082.283822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2082.288732] LustreError: Skipped 1 previous similar message [ 2082.290751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2082.301073] Lustre: Skipped 3 previous similar messages [ 2082.314549] Lustre: Skipped 4 previous similar messages [ 2088.806310] Lustre: server umount lustre-OST0000 complete [ 2096.616278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2096.648500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.684276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2096.687089] Lustre: Skipped 2 previous similar messages [ 2107.875829] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2108.521605] Lustre: server umount lustre-MDT0000 complete [ 2112.678951] LustreError: 64448:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786307153 with bad export cookie 14916868678367812077 [ 2112.686947] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2112.688323] LustreError: 64448:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2113.267228] Lustre: server umount lustre-MDT0001 complete [ 2127.777106] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2131.551210] Key type lgssc unregistered [ 2131.941756] LNet: 68238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.958070] LNetError: 68238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2131.992759] LNet: Removed LNI 192.168.202.102@tcp [ 2133.335172] Key type .llcrypt unregistered [ 2133.341160] Key type ._llcrypt unregistered [ 2164.675203] Key type ._llcrypt registered [ 2164.680712] Key type .llcrypt registered [ 2164.838995] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2166.535525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2166.744404] alg: No test for adler32 (adler32-zlib) [ 2168.152403] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 2168.570544] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2170.327519] Key type lgssc registered [ 2172.340241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2182.686743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2182.701239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2184.070636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2188.397455] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2193.401693] LustreError: 69656:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2193.424478] LustreError: 69656:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2198.303360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2198.496837] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2198.863954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2204.050440] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2211.800389] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2217.681168] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2226.043723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2226.824712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2232.959098] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2240.652639] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2249.913663] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2258.915696] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.918240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2258.933069] Lustre: Skipped 2 previous similar messages [ 2258.953722] Lustre: Skipped 2 previous similar messages [ 2264.037175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2264.045742] Lustre: Skipped 2 previous similar messages [ 2264.657560] Lustre: server umount lustre-MDT0000 complete [ 2268.627792] LustreError: 69636:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786307309 with bad export cookie 9126107682527588426 [ 2268.637284] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2268.637798] LustreError: 69636:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2268.973742] Lustre: server umount lustre-MDT0001 complete [ 2283.399524] Lustre: server umount lustre-OST0000 complete [ 2312.646376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2313.000046] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2313.038301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2313.249034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2318.027921] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2326.242836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.413135] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2326.463929] Lustre: 74527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2327.106669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2331.178247] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2339.112653] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2346.318294] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2354.227672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2354.364507] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2354.722772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2361.318797] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2370.484965] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2372.274818] Lustre: 77101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2374.120103] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2383.336428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2383.339850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2383.370708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2383.844875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2386.152558] Lustre: server umount lustre-OST0000 complete [ 2393.569467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2393.574108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2393.591796] Lustre: Skipped 1 previous similar message [ 2393.597694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2393.608623] Lustre: Skipped 1 previous similar message [ 2403.305120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2403.309185] Lustre: Skipped 5 previous similar messages [ 2404.319187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2404.616618] Lustre: server umount lustre-MDT0000 complete [ 2408.417195] LustreError: 77283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.443149] LustreError: 77283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2409.368367] LustreError: 74518:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786307450 with bad export cookie 9126107682527589273 [ 2409.371049] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2409.377423] LustreError: 74518:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2409.821896] Lustre: server umount lustre-MDT0001 complete [ 2424.696777] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2428.722885] Key type lgssc unregistered [ 2429.274605] LNet: 78316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2429.278858] LNetError: 78316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2429.289718] LNet: Removed LNI 192.168.202.102@tcp [ 2430.807090] Key type .llcrypt unregistered [ 2430.810131] Key type ._llcrypt unregistered [ 2462.603411] Key type ._llcrypt registered [ 2462.605877] Key type .llcrypt registered [ 2462.729562] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2464.148203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2464.180210] alg: No test for adler32 (adler32-zlib) [ 2465.373505] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 2465.688113] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2467.351288] Key type lgssc registered [ 2468.530739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2478.192520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2478.209743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.824057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2485.604483] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2489.319621] LustreError: 79732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.344462] LustreError: 79732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2494.441520] LustreError: 79733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.908158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2497.787839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2502.885531] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2512.529345] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2520.358904] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2529.280054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2529.812992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.078733] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2546.089650] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2556.075590] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2557.119814] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2558.178524] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2561.240505] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2561.244883] Lustre: Skipped 2 previous similar messages [ 2565.398076] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2565.404717] Lustre: Skipped 3 previous similar messages [ 2574.027899] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2574.035442] Lustre: Skipped 8 previous similar messages [ 2590.383404] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2590.395233] Lustre: Skipped 15 previous similar messages [ 2622.933979] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2622.936673] Lustre: Skipped 27 previous similar messages [ 2680.850533] Lustre: 87847:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2684.081698] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 16:35:23 (1786307723) [ 2685.779820] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2688.094428] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 16:35:27 (1786307727) [ 2695.660346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2695.662809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2695.681978] Lustre: Skipped 1 previous similar message [ 2695.702535] Lustre: Skipped 1 previous similar message [ 2699.676844] Lustre: server umount lustre-OST0000 complete [ 2705.889860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2705.910235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2711.012163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2711.038931] Lustre: Skipped 2 previous similar messages [ 2711.228452] Lustre: server umount lustre-MDT0000 complete [ 2715.123941] LustreError: 79714:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786307755 with bad export cookie 9464767993031037268 [ 2715.127895] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2715.134177] LustreError: 79714:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2715.931410] Lustre: server umount lustre-MDT0001 complete [ 2731.015217] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2734.435556] Key type lgssc unregistered [ 2734.796901] LNet: 89100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2734.806096] LNetError: 89100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2734.848338] LNet: Removed LNI 192.168.202.102@tcp [ 2735.971319] Key type .llcrypt unregistered [ 2735.976870] Key type ._llcrypt unregistered [ 2742.021221] Key type ._llcrypt registered [ 2742.023546] Key type .llcrypt registered [ 2742.145509] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2747.875480] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules [ 2749.495931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2749.536330] alg: No test for adler32 (adler32-zlib) [ 2750.640980] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 2750.963260] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2752.751215] Key type lgssc registered [ 2754.332822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2755.788213] LustreError: 90315:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2777.262741] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2777.308793] LustreError: 90315:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2790.408470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2790.464389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2792.342883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2798.591952] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2801.646315] LustreError: 91327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.667583] LustreError: 91327:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2806.767618] LustreError: 91328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.759794] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 16:37:29 (1786307849) [ 2811.874412] LustreError: 91327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2816.997089] LustreError: 91328:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2817.263573] LustreError: 91360:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2817.270564] LustreError: 91360:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2817.274496] LustreError: 91360:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 25, retries 0, failed: rc = -5 [ 2817.729770] Lustre: server umount lustre-MDT0000 complete [ 2833.834799] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 2837.436529] Key type lgssc unregistered [ 2837.791438] LNet: 92859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2837.801271] LNetError: 92859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2837.838251] LNet: Removed LNI 192.168.202.102@tcp [ 2839.069191] Key type .llcrypt unregistered [ 2839.072077] Key type ._llcrypt unregistered [ 2863.122047] Key type ._llcrypt registered [ 2863.123623] Key type .llcrypt registered [ 2863.241714] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 2865.630958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2865.708929] alg: No test for adler32 (adler32-zlib) [ 2866.810778] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 2867.214873] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 2868.919567] Key type lgssc registered [ 2870.666209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2879.998110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2880.028981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2881.587447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2887.342542] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2891.744638] LustreError: 94078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2891.754248] LustreError: 94078:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2896.866816] LustreError: 94079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2897.445419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2898.504216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2904.514581] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2912.624461] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2919.516953] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2929.227539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2929.880310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.462434] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 2947.903586] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3010.912592] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 16:40:49 (1786308049) [ 3012.676790] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 3014.931792] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 16:40:53 (1786308053) [ 3020.770617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3020.781253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3020.802851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3023.331527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3023.340996] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3026.473852] Lustre: server umount lustre-MDT0000 complete [ 3028.459906] LustreError: 94079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3028.493415] LustreError: 94079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3030.766916] LustreError: 94820:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786308071 with bad export cookie 5924122818649740129 [ 3030.782682] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3031.198922] Lustre: server umount lustre-MDT0001 complete [ 3046.896172] Lustre: server umount lustre-OST0000 complete [ 3061.820118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.430520] LustreError: 98387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3062.733608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3067.874266] LustreError: 98388:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3069.551820] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3078.113575] LustreError: 98387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3078.149095] LustreError: 98387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3079.687712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3080.501528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3086.530492] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3096.522499] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3103.353812] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3119.641669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3119.807631] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3119.813218] LustreError: 98376:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3119.834326] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3119.840405] LustreError: 100473:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3119.846470] LustreError: 100473:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3119.855355] LustreError: 100473:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3119.918564] Lustre: server umount lustre-OST0000 complete [ 3119.921562] LustreError: 100473:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3129.708101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3129.907059] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3129.915386] LustreError: 98377:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3129.943134] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3129.956908] LustreError: 100940:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 3129.974887] LustreError: 100940:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 3129.988744] LustreError: 100940:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 3130.054524] Lustre: server umount lustre-OST0000 complete [ 3130.059440] LustreError: 100940:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3140.893241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3141.040560] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3141.043896] Lustre: Found index 0 for lustre-OST0000, updating log [ 3141.048394] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3141.112699] Lustre: lustre-OST0000: new disk, initializing [ 3141.118311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3141.122609] Lustre: 101477:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3141.373117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3149.710688] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3154.808272] Lustre: Failing over lustre-OST0000 [ 3154.926375] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3154.940763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.966645] Lustre: Skipped 1 previous similar message [ 3154.979824] LustreError: 101462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3155.189949] Lustre: server umount lustre-OST0000 complete [ 3162.084334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3162.097066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3162.119092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3162.129290] Lustre: Skipped 2 previous similar messages [ 3167.156053] Lustre: server umount lustre-MDT0000 complete [ 3171.799740] LustreError: 100492:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786308212 with bad export cookie 5924122818649742873 [ 3171.811406] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3171.815550] LustreError: 100492:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3172.253403] Lustre: server umount lustre-MDT0001 complete [ 3184.391605] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 16:43:43 (1786308223) [ 3206.355132] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3218.775210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.442976] LustreError: 103823:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3219.483892] LustreError: 103823:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3219.646647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3225.152978] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3235.394332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3241.644913] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3245.358417] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3252.780622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3253.809654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3253.822515] Lustre: Skipped 1 previous similar message [ 3253.838810] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3260.978889] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3261.027936] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3261.028300] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3261.593482] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3272.620940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3272.730305] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3272.855763] Lustre: lustre-OST0001: new disk, initializing [ 3272.860628] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3272.868709] Lustre: 106141:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3281.038740] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3283.085188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3283.101198] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3283.298665] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3290.338949] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3301.540691] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3308.016131] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 3308.029062] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.051596] Lustre: Skipped 1 previous similar message [ 3308.056808] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3308.059365] Lustre: Skipped 2 previous similar messages [ 3312.701425] Lustre: server umount lustre-MDT0001 complete [ 3314.152972] LustreError: 103818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.178370] LustreError: 103818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3326.762955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.895734] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3326.914156] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3327.028446] Lustre: 103813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3328.399548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3328.420969] Lustre: Skipped 1 previous similar message [ 3328.672959] LustreError: 107665:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3328.686612] LustreError: 107665:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3328.704890] LustreError: 103816:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3328.863290] LustreError: 107682:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.102@tcp: cfg command failed: rc = -17 [ 3328.876946] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3328.876946] [ 3333.650464] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3334.291865] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3346.749751] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 16:46:25 (1786308385) [ 3367.240569] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3385.878465] Lustre: 109758:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3410.716821] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3414.616414] Lustre: 110894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3507.680412] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3507.684395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3507.700953] Lustre: Skipped 4 previous similar messages [ 3507.720180] Lustre: Skipped 7 previous similar messages [ 3509.493615] Lustre: server umount lustre-MDT0000 complete [ 3512.818882] LustreError: 103820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3512.840780] LustreError: 103820:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 3514.693222] LustreError: 106140:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786308555 with bad export cookie 5924122818649743867 [ 3514.698685] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3514.720156] LustreError: 106140:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3515.101691] Lustre: server umount lustre-MDT0001 complete [ 3529.549983] Lustre: server umount lustre-OST0000 complete [ 3543.958621] Lustre: server umount lustre-OST0001 complete [ 3549.616413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3560.250700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3570.312387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3583.097789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3607.652677] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3619.916295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3619.940198] Lustre: lustre-MDT0000: reset Object Index mappings [ 3620.644487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3625.700659] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3635.563782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3635.607306] Lustre: lustre-MDT0001: reset Object Index mappings [ 3641.406316] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3644.894719] Lustre: 115997:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3651.786964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3657.240642] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3660.587224] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3665.743367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3668.068506] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3674.629389] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3674.670588] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3678.125123] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3688.919976] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3694.782303] Lustre: 117877:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3752.742631] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 16:53:11 (1786308791) [ 3779.555720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3779.570171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3779.588446] Lustre: Skipped 1 previous similar message [ 3779.597609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3785.047810] Lustre: server umount lustre-MDT0000 complete [ 3787.233375] LustreError: 115587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3787.261499] LustreError: 115587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 3788.753526] LustreError: 114829:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786308829 with bad export cookie 5924122818649790704 [ 3788.765518] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3788.778980] LustreError: 114829:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3795.353406] Lustre: server umount lustre-MDT0001 complete [ 3814.369900] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3814.476789] Lustre: server umount lustre-OST0000 complete [ 3824.754213] Lustre: server umount lustre-OST0001 complete [ 3832.143689] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 3845.267691] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3888.550491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3888.759993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3888.796565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3888.868199] Lustre: lustre-MDT0000: new disk, initializing [ 3888.958726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3888.964766] Lustre: Skipped 3 previous similar messages [ 3888.976245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3893.378442] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3901.106631] Lustre: server umount lustre-MDT0000 complete [ 3911.867901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.160672] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3912.180952] Lustre: 122153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3912.200126] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3918.036777] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3925.254233] Lustre: server umount lustre-MDT0000 complete [ 3935.618260] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 16:56:14 (1786308974) [ 3937.372297] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3939.720199] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 16:56:18 (1786308978) [ 3957.725480] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 3967.772860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3971.913123] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3979.875505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3980.072446] Lustre: 124203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3980.148511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3980.155395] Lustre: Skipped 1 previous similar message [ 3980.241787] Lustre: lustre-MDT0001: new disk, initializing [ 3980.404308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3980.420106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3985.596665] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 3990.326682] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3997.833014] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3998.081455] Lustre: lustre-OST0000: new disk, initializing [ 3998.084890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3998.091707] Lustre: 125781:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4000.165596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4000.177124] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4000.288274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4004.771531] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4016.237108] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4016.488965] Lustre: lustre-OST0001: new disk, initializing [ 4016.494296] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4016.501920] Lustre: 126652:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4018.141103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4018.163540] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4018.252507] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4022.930440] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4033.296704] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4037.323657] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4446.197338] Lustre: 133078:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4446.217971] Lustre: 133078:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4446.228820] Lustre: 133078:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4446.235291] Lustre: 133078:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4446.238182] Lustre: 133078:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4446.246718] Lustre: 133078:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4762.189344] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 17:10:00 (1786309800) [ 4781.028957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4781.032918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4781.049067] Lustre: Skipped 7 previous similar messages [ 4781.051641] Lustre: Skipped 9 previous similar messages [ 4785.640754] Lustre: server umount lustre-MDT0000 complete [ 4786.152304] LustreError: 127465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4786.188749] LustreError: 127465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4790.230099] LustreError: 125780:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786309831 with bad export cookie 5924122818649860137 [ 4790.245069] LustreError: 125780:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4790.246125] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4791.273404] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4791.280805] Lustre: Skipped 2 previous similar messages [ 4791.284651] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4791.295961] Lustre: Skipped 3 previous similar messages [ 4795.300288] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4795.312303] Lustre: Skipped 1 previous similar message [ 4796.966356] Lustre: server umount lustre-MDT0001 complete [ 4807.727350] Lustre: server umount lustre-OST0000 complete [ 4818.396331] Lustre: server umount lustre-OST0001 complete [ 4826.216389] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 4839.227536] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 4886.105113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4886.362256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4886.398366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4886.519722] Lustre: lustre-MDT0000: new disk, initializing [ 4886.629450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4886.632367] Lustre: Skipped 5 previous similar messages [ 4886.674657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4893.726209] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4905.180770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4905.367261] Lustre: 136139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4905.453776] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4905.457253] Lustre: Skipped 1 previous similar message [ 4905.597938] Lustre: lustre-MDT0001: new disk, initializing [ 4905.754613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4905.767991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4910.577962] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4921.782782] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4929.025853] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4936.706528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4936.956368] Lustre: lustre-OST0000: new disk, initializing [ 4936.958772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4936.962788] Lustre: 138109:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4938.764740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4938.776740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4938.893139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4943.827665] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 4954.017575] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4963.244671] Lustre: lustre-MDD0000: changelog on [ 4965.909711] Lustre: lustre-MDD0001: changelog on [ 6975.121775] Lustre: lustre-MDD0001: changelog off [ 6977.831537] Lustre: 136147:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6978.689783] Lustre: lustre-MDD0000: changelog off [ 6986.733851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6986.737938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6986.749448] LustreError: Skipped 1 previous similar message [ 6986.757074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6986.785395] Lustre: Skipped 1 previous similar message [ 6989.800355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6989.808409] Lustre: Skipped 2 previous similar messages [ 6992.753956] Lustre: server umount lustre-OST0000 complete [ 7000.033044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7000.038403] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7000.040615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7000.062746] Lustre: Skipped 1 previous similar message [ 7004.096839] Lustre: server umount lustre-MDT0000 complete [ 7005.168246] LustreError: 136147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7005.194349] LustreError: 136147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7008.884433] LustreError: 136132:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312049 with bad export cookie 5924122818650683316 [ 7008.888194] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7008.912700] LustreError: 136132:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7009.284512] Lustre: server umount lustre-MDT0001 complete [ 7024.740899] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7028.225943] Key type lgssc unregistered [ 7028.573232] LNet: 147260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7028.591325] LNetError: 147260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7028.615164] LNet: Removed LNI 192.168.202.102@tcp [ 7029.652263] Key type .llcrypt unregistered [ 7029.654224] Key type ._llcrypt unregistered [ 7052.156866] Key type ._llcrypt registered [ 7052.163136] Key type .llcrypt registered [ 7052.334649] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 7073.217938] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7074.532484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7074.801263] alg: No test for adler32 (adler32-zlib) [ 7075.984541] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 7076.364657] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 7078.079485] Key type lgssc registered [ 7079.546037] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7122.114490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7122.181651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7123.468556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7123.499938] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7123.624835] Lustre: lustre-MDT0000: new disk, initializing [ 7123.700139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7123.719620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7129.676567] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7142.738285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.003392] Lustre: 150462:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7143.071236] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7143.079564] Lustre: Skipped 1 previous similar message [ 7143.167815] Lustre: lustre-MDT0001: new disk, initializing [ 7143.339588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7143.406094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7143.417090] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7148.711905] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7160.459748] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7168.511160] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7178.326615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7178.760227] Lustre: lustre-OST0000: new disk, initializing [ 7178.770651] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7178.776788] Lustre: 152435:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7178.932437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7185.995693] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7187.562039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7187.570784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7187.699027] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7197.119731] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7204.028337] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_module obdecho/obdecho [ 7206.300383] LustreError: 153437:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7206.329126] LustreError: 153437:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7208.604080] LustreError: 153534:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7223.779346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7223.783800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7223.796272] Lustre: Skipped 2 previous similar messages [ 7223.799151] Lustre: Skipped 1 previous similar message [ 7228.293154] Lustre: server umount lustre-MDT0000 complete [ 7232.577409] LustreError: 151261:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312273 with bad export cookie 8751832370966932362 [ 7232.587224] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7232.602043] LustreError: 151261:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7233.017269] Lustre: server umount lustre-MDT0001 complete [ 7247.850627] Lustre: server umount lustre-OST0000 complete [ 7271.226836] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 7284.270302] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7326.509940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7327.031804] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7327.081785] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7327.186688] Lustre: lustre-MDT0000: new disk, initializing [ 7327.339192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7327.360212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7333.311145] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7345.619047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7345.807675] Lustre: 157253:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7345.912159] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7345.924442] Lustre: Skipped 1 previous similar message [ 7346.034252] Lustre: lustre-MDT0001: new disk, initializing [ 7346.138476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7346.189546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7346.202467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7353.112893] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7367.188529] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7374.636849] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7383.002784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7383.280140] Lustre: lustre-OST0000: new disk, initializing [ 7383.283813] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7383.290662] Lustre: 159226:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7383.398948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7389.575943] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7391.270965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7391.277507] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7391.401300] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7399.885061] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7405.330595] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7405.596554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7411.760227] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7412.068498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7416.800274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7416.812840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7416.856801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7416.863187] Lustre: Skipped 1 previous similar message [ 7417.312292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7421.158831] Lustre: server umount lustre-OST0000 complete [ 7427.560281] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7427.580286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7427.609186] Lustre: Skipped 2 previous similar messages [ 7427.623727] Lustre: Skipped 2 previous similar messages [ 7432.675806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7432.693436] Lustre: Skipped 2 previous similar messages [ 7437.793119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7437.797325] Lustre: Skipped 1 previous similar message [ 7441.375125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7441.732867] Lustre: server umount lustre-MDT0000 complete [ 7442.914381] LustreError: 158272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7442.941150] LustreError: 158272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 7446.464965] LustreError: 157244:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312487 with bad export cookie 8751832370966933321 [ 7446.476204] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7446.478878] LustreError: 157244:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7446.939052] Lustre: server umount lustre-MDT0001 complete [ 7451.823333] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 17:54:50 (1786312490) [ 7454.520935] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7457.422315] Lustre: DEBUG MARKER: == conf-sanity test 138: MDT mount failure must not LBUG destroying recovery exports ========================================================== 17:54:55 (1786312495) [ 7481.827024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7482.466785] LustreError: 162199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7482.527552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7487.973720] LustreError: 162200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7488.029845] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7493.091102] LustreError: 162199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7498.210941] LustreError: 162200:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7502.214104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7510.373890] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7521.447960] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7529.327343] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7537.606906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7537.965082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7537.968959] Lustre: Skipped 1 previous similar message [ 7546.157679] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7554.795740] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7561.899992] Lustre: *** cfs_fail_loc=718, val=0*** [ 7562.450887] Lustre: server umount lustre-MDT0000 complete [ 7563.746913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7563.753546] LustreError: 162196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7563.773877] Lustre: Skipped 2 previous similar messages [ 7563.850880] LustreError: 162196:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7566.855352] Lustre: *** cfs_fail_loc=718, val=0*** [ 7566.858507] LustreError: 162935:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312607 with bad export cookie 8751832370966934203 [ 7566.863947] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7566.869877] Lustre: Skipped 3 previous similar messages [ 7566.916832] LustreError: 162935:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7567.808667] Lustre: server umount lustre-MDT0001 complete [ 7579.127527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7585.247471] Lustre: 148686:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786312609/real 1786312609] req@00000000dbb85007 x1873084014514560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786312625 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7585.297361] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7587.360080] Lustre: 148686:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786312612/real 1786312612] req@0000000029b52c02 x1873084014514688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786312628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7589.344049] Lustre: 148688:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786312614/real 1786312614] req@000000003b2b3492 x1873084014514816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786312630 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7591.802497] Lustre: *** cfs_fail_loc=723, val=0*** [ 7591.806136] Lustre: Skipped 1 previous similar message [ 7591.979547] LustreError: 165560:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-122) [ 7591.983152] LustreError: 165560:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.102@tcp: cfg command failed: rc = -122 [ 7591.990538] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 7591.990538] [ 7592.001652] LustreError: MGC192.168.202.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -122. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 7592.009878] LustreError: 165530:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -122 [ 7592.018362] LustreError: 165530:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -122 [ 7592.028992] LustreError: 165530:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 7592.038831] LustreError: 165530:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 7592.167052] Lustre: server umount lustre-MDT0000 complete [ 7592.173658] LustreError: 165530:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -122 [ 7592.476299] Lustre: 148689:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786312617/real 1786312617] req@0000000094ccdc61 x1873084014515072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786312633 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7597.613060] Lustre: 148686:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786312622/real 1786312622] req@00000000aa0f2d5b x1873084014515200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786312638 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7601.036450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7601.319930] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7601.339030] LustreError: 162191:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 7601.809041] LustreError: 164034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7601.825600] LustreError: 164034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7601.947541] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7603.949751] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7605.741014] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7606.896591] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7616.146782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7616.391695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 7616.524615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7616.527416] Lustre: Skipped 1 previous similar message [ 7616.577772] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7617.383856] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7621.631604] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7621.719251] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7621.982433] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7631.756809] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7633.780700] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7642.112354] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7644.466684] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7649.693773] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 7663.076693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7663.078965] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7663.082185] LustreError: Skipped 1 previous similar message [ 7663.085516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7663.103392] Lustre: Skipped 1 previous similar message [ 7669.420728] Lustre: server umount lustre-OST0000 complete [ 7673.316968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7679.297527] Lustre: server umount lustre-MDT0000 complete [ 7683.051305] LustreError: 165974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7683.086320] LustreError: 165974:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 7683.586035] LustreError: 162180:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312724 with bad export cookie 8751832370966935155 [ 7683.603637] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7683.603881] LustreError: 162180:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7683.894662] Lustre: server umount lustre-MDT0001 complete [ 7696.786789] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7699.509606] Key type lgssc unregistered [ 7699.866457] LNet: 169040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7699.872591] LNetError: 169040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7700.909637] LNet: Removed LNI 192.168.202.102@tcp [ 7701.939714] Key type .llcrypt unregistered [ 7701.949879] Key type ._llcrypt unregistered [ 7741.945654] Key type ._llcrypt registered [ 7741.948485] Key type .llcrypt registered [ 7742.094771] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7743.881096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7743.895980] alg: No test for adler32 (adler32-zlib) [ 7745.056344] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 7745.437029] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 7747.154141] Key type lgssc registered [ 7748.966601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7757.272844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7757.296550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7758.875859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7763.959613] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7769.057625] LustreError: 170909:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7769.098740] LustreError: 170909:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7773.892343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7774.179350] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7774.310666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7778.731875] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7787.731116] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7794.965441] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7803.008185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7803.441481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7804.482269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7809.828652] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7818.259659] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7826.965747] Lustre: *** cfs_fail_loc=718, val=0*** [ 7827.569635] Lustre: server umount lustre-MDT0000 complete [ 7828.969618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7828.978069] LustreError: 170904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7828.985619] Lustre: Skipped 1 previous similar message [ 7829.022643] LustreError: 170904:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7831.433392] LustreError: 170904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7831.559081] Lustre: *** cfs_fail_loc=718, val=0*** [ 7831.561801] Lustre: Skipped 3 previous similar messages [ 7831.567504] LustreError: 170891:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312872 with bad export cookie 12198485859914294857 [ 7831.569720] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7831.577119] LustreError: 170891:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7831.962219] Lustre: server umount lustre-MDT0001 complete [ 7846.115586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7849.443209] Lustre: 170223:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786312874/real 1786312874] req@000000004bd03432 x1873084715922304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786312890 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7849.460520] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7849.469360] Lustre: Skipped 1 previous similar message [ 7851.487318] Lustre: 170221:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786312876/real 1786312876] req@0000000039bb99bd x1873084715922432/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786312892 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7854.623750] Lustre: 170221:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786312879/real 1786312879] req@00000000f85b6b2c x1873084715922560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786312895 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7856.671592] LustreError: 170220:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000019d48482 x1873084715923456/t0(0) o250->MGC192.168.202.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7857.198941] Lustre: *** cfs_fail_loc=727, val=0*** [ 7857.295439] LustreError: 174327:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-57) [ 7857.312132] LustreError: 174327:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.102@tcp: cfg command failed: rc = -57 [ 7857.326701] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 7857.326701] [ 7857.343976] LustreError: MGC192.168.202.102@tcp: Confguration from log lustre-MDT0000 failed from MGS -57. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 7857.371359] LustreError: 174298:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -57 [ 7857.400223] LustreError: 174298:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -57 [ 7857.408277] LustreError: 174298:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 7857.413342] LustreError: 174298:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 7857.485342] Lustre: server umount lustre-MDT0000 complete [ 7857.489566] LustreError: 174298:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -57 [ 7858.463204] Lustre: 170223:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786312882/real 1786312882] req@000000002ce66b97 x1873084715922816/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786312898 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7863.648631] Lustre: 170223:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786312888/real 1786312888] req@00000000eec11cc9 x1873084715922944/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786312904 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7868.627853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7868.740258] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7868.753189] LustreError: 170901:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 7868.760076] LustreError: 174819:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.102@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000888ed1aa) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 7869.029967] LustreError: 172742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7869.099423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7869.146116] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7870.147653] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7871.407415] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7874.296811] LustreError: 172743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7874.322137] LustreError: 172743:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7875.740126] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7883.133020] LustreError: 174830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7883.158535] LustreError: 174830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7886.695399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7886.922622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 7887.046298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7887.096528] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 7888.230109] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7892.471238] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7892.491382] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7892.579730] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7892.775542] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 7901.652721] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7904.151811] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7911.060480] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7912.984739] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7917.559181] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 7933.417212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7933.429090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7933.430269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7933.442765] Lustre: Skipped 1 previous similar message [ 7937.582123] Lustre: server umount lustre-OST0000 complete [ 7943.649988] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7943.661325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7943.661336] Lustre: Skipped 1 previous similar message [ 7943.662296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7948.325876] Lustre: server umount lustre-MDT0000 complete [ 7948.769208] LustreError: 174833:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7948.801460] LustreError: 174833:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7952.878852] LustreError: 170890:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786312993 with bad export cookie 12198485859914296082 [ 7952.888618] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7952.891915] LustreError: 170890:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7953.402151] Lustre: server umount lustre-MDT0001 complete [ 7966.375572] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 7969.444806] Key type lgssc unregistered [ 7969.685214] LNet: 177907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7969.691547] LNetError: 177907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7969.713080] LNet: Removed LNI 192.168.202.102@tcp [ 7970.944252] Key type .llcrypt unregistered [ 7970.946308] Key type ._llcrypt unregistered [ 7996.985631] Key type ._llcrypt registered [ 7996.991650] Key type .llcrypt registered [ 7997.086479] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 7998.348046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7998.484948] alg: No test for adler32 (adler32-zlib) [ 7999.541892] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 7999.773810] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 8001.464350] Key type lgssc registered [ 8002.919851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8011.386946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8011.406883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8012.777637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8017.523864] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8023.013657] LustreError: 179225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8023.060291] LustreError: 179225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8026.285404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8026.720469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8031.384809] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8040.659029] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8047.846437] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8055.457165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8055.812161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8057.850649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8062.540317] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8070.601967] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8078.742742] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 8082.407248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8082.413244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8082.423935] Lustre: Skipped 2 previous similar messages [ 8082.439731] Lustre: Skipped 2 previous similar messages [ 8083.821372] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [ 8084.453166] Lustre: server umount lustre-MDT0000 complete [ 8087.535544] LustreError: 179225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8087.563958] LustreError: 179225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8088.389826] LustreError: 179205:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313129 with bad export cookie 4204706027040406892 [ 8088.400259] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8088.402388] LustreError: 179205:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8088.937483] LustreError: 180465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8088.992842] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.2@tcp (stopping) [ 8092.641453] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8092.643493] LustreError: 179221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8093.665181] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8093.675040] Lustre: Skipped 1 previous similar message [ 8094.814196] Lustre: server umount lustre-MDT0001 complete [ 8101.401919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 8116.308315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8116.568390] LustreError: 183067:0:(llog.c:1655:llog_backup()) MGC192.168.202.102@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 8116.573934] Lustre: 183067:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.102@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 8118.242573] LustreError: 183072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8118.267831] LustreError: 183072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8128.480428] LustreError: 183074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8128.498049] LustreError: 183074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 8146.279929] LustreError: 183073:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8146.292264] LustreError: 183073:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 8177.119412] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 8177.128798] Lustre: 183067:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.202.102@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 8179.681262] LustreError: 183072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8179.693920] LustreError: 183072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 24 previous similar messages [ 8218.108861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8222.858479] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8223.210360] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8232.307128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8233.476765] LustreError: 178532:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c3548683 x1873084982279808/t0(0) o250->MGC192.168.202.102@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8233.905656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8233.991977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:193) [ 8239.094507] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8239.165823] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8248.404740] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8250.470632] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8257.560546] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8259.478551] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8280.038195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8280.040465] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8280.059511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8280.059519] Lustre: Skipped 1 previous similar message [ 8280.092653] Lustre: Skipped 1 previous similar message [ 8284.055117] Lustre: server umount lustre-MDT0000 complete [ 8285.155690] LustreError: 183072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8285.183832] LustreError: 183072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 39 previous similar messages [ 8287.771187] LustreError: 179206:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313328 with bad export cookie 4204706027040413563 [ 8287.783418] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8287.790903] LustreError: 179206:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8288.298611] Lustre: server umount lustre-MDT0001 complete [ 8303.371612] Lustre: server umount lustre-OST0000 complete [ 8327.566077] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 8341.798939] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8384.576480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8384.878964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8384.958150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8385.092317] Lustre: lustre-MDT0000: new disk, initializing [ 8385.257508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8385.281872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8389.794625] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8400.638151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8400.817585] Lustre: 188191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8400.957656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8400.978678] Lustre: Skipped 1 previous similar message [ 8401.097042] Lustre: lustre-MDT0001: new disk, initializing [ 8401.295437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8401.317331] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8405.982746] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8416.200715] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8422.942759] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8430.004195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8430.322517] Lustre: lustre-OST0000: new disk, initializing [ 8430.325580] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8430.332500] Lustre: 190160:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8430.421414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8430.433489] Lustre: Skipped 1 previous similar message [ 8436.738802] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8438.298527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8438.306868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8438.425802] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8446.316131] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8451.172639] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8451.484948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8456.785506] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8457.111784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8463.840815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8463.847181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8463.858324] Lustre: Skipped 1 previous similar message [ 8463.874416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8463.882181] Lustre: Skipped 2 previous similar messages [ 8465.775315] Lustre: server umount lustre-OST0000 complete [ 8473.057288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8473.065636] LustreError: Skipped 1 previous similar message [ 8473.072636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8473.086258] Lustre: Skipped 1 previous similar message [ 8484.319246] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8484.498364] Lustre: server umount lustre-MDT0000 complete [ 8484.846200] LustreError: 188995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8484.883796] LustreError: 188995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8489.697069] LustreError: 188183:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313530 with bad export cookie 4204706027040430622 [ 8489.699304] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8489.718719] LustreError: 188183:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8490.064770] Lustre: server umount lustre-MDT0001 complete [ 8500.333144] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 18:12:19 (1786313539) [ 8508.352521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8508.957177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8513.977873] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8522.674429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8528.225471] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8539.089961] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8545.816858] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8553.555904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8553.958833] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 8553.968302] Lustre: Skipped 7 previous similar messages [ 8562.074879] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8570.367610] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8580.391826] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 18:13:39 (1786313619) [ 8584.677521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8584.682280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8584.690372] Lustre: Skipped 1 previous similar message [ 8590.748342] Lustre: server umount lustre-OST0000 complete [ 8602.184282] Lustre: server umount lustre-MDT0000 complete [ 8607.194799] LustreError: 192554:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313647 with bad export cookie 4204706027040431504 [ 8607.203956] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8607.209248] LustreError: 192554:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8607.873907] Lustre: server umount lustre-MDT0001 complete [ 8620.770208] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 8623.895535] Key type lgssc unregistered [ 8624.212848] LNet: 196308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8624.236058] LNetError: 196308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8624.259779] LNet: Removed LNI 192.168.202.102@tcp [ 8625.237699] Key type .llcrypt unregistered [ 8625.241790] Key type ._llcrypt unregistered [ 8650.180613] Key type ._llcrypt registered [ 8650.182675] Key type .llcrypt registered [ 8650.282353] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8651.638944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8651.846974] alg: No test for adler32 (adler32-zlib) [ 8652.949261] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 8653.171686] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 8654.833133] Key type lgssc registered [ 8656.274258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8664.560231] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8664.582401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8680.159504] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8685.279832] LustreError: 197553:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.102@tcp: failed processing log, type 4: rc = -110 [ 8690.399849] LustreError: 197553:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8690.411524] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8690.428059] LustreError: MGC192.168.202.102@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8690.446268] LustreError: 197553:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8690.475164] LustreError: 197553:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 8690.485531] LustreError: 197553:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8690.575393] Lustre: server umount lustre-OST0000 complete [ 8690.577397] LustreError: 197553:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8700.297876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8701.834749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8706.613830] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8712.177147] LustreError: 198053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8712.190469] LustreError: 198053:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8716.629383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8717.138097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8721.937494] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8730.734268] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8737.757779] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8745.111753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8745.296647] LustreError: 199870:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8745.308349] Lustre: 199870:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.102@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8745.437475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8752.725262] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8761.327040] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8766.956105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8766.980928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8766.989842] Lustre: Skipped 2 previous similar messages [ 8772.067423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8772.077628] Lustre: Skipped 2 previous similar messages [ 8777.186481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8777.218770] Lustre: Skipped 2 previous similar messages [ 8778.207314] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8780.588781] Lustre: server umount lustre-MDT0000 complete [ 8782.306120] LustreError: 198048:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8782.340156] LustreError: 198048:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8785.661789] LustreError: 198034:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786313826 with bad export cookie 10895737202044549 [ 8785.664701] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8785.676687] LustreError: 198034:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8786.255367] Lustre: server umount lustre-MDT0001 complete [ 8802.035464] Lustre: server umount lustre-OST0000 complete [ 8814.379485] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 8826.738238] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 8869.203876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8869.495926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8869.532354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8869.616375] Lustre: lustre-MDT0000: new disk, initializing [ 8869.677808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8869.712676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8874.025208] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8885.681705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8885.819683] Lustre: 203397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8885.872277] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8885.875354] Lustre: Skipped 1 previous similar message [ 8886.045991] Lustre: lustre-MDT0001: new disk, initializing [ 8886.206626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8886.241049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8886.264461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8891.324896] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8903.888278] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8911.090508] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8920.214057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8920.580989] Lustre: lustre-OST0000: new disk, initializing [ 8920.584748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8920.591948] Lustre: 205369:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8920.683872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8928.979081] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 8929.317371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8929.320699] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8929.485952] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8939.002398] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8945.606549] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8945.901979] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8951.498878] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8951.809871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8954.853185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8954.868778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8954.896811] Lustre: Skipped 2 previous similar messages [ 8954.916787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8955.366809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8960.421921] Lustre: server umount lustre-OST0000 complete [ 8965.601293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8965.604676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8965.611495] Lustre: Skipped 2 previous similar messages [ 8965.623872] Lustre: Skipped 3 previous similar messages [ 8971.084514] Lustre: server umount lustre-MDT0000 complete [ 8975.221698] LustreError: 203391:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314015 with bad export cookie 10895737202045270 [ 8975.231257] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8975.238786] LustreError: 203391:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8975.771749] Lustre: server umount lustre-MDT0001 complete [ 8986.723938] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 18:20:25 (1786314025) [ 9007.194207] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [ 9010.417282] Key type lgssc unregistered [ 9010.788513] LNet: 208318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9010.802566] LNetError: 208318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9010.836956] LNet: Removed LNI 192.168.202.102@tcp [ 9012.005756] Key type .llcrypt unregistered [ 9012.007381] Key type ._llcrypt unregistered [ 9032.575718] Key type ._llcrypt registered [ 9032.579200] Key type .llcrypt registered [ 9032.749198] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9034.167411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9034.418031] alg: No test for adler32 (adler32-zlib) [ 9035.600548] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [ 9036.022244] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [ 9037.807190] Key type lgssc registered [ 9039.043079] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9047.811652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9047.843613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9049.451745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9054.385784] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9059.809671] LustreError: 209538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9059.827588] LustreError: 209538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9060.839525] LustreError: 209539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9063.740146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9063.979392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9069.436325] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9077.616244] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9084.001292] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9092.620063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9092.945255] Lustre: *** cfs_fail_loc=131e, val=0*** [ 9092.952654] LustreError: 211402:0:(llog.c:1696:llog_backup()) MGC192.168.202.102@tcp: failed to backup log lustre-OST0000: rc = -28 [ 9092.956783] Lustre: 211402:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.102@tcp: can't backup local config lustre-OST0000: rc = -28 [ 9093.047839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9099.525556] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9108.277457] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9113.573633] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9113.584382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9113.601711] Lustre: Skipped 2 previous similar messages [ 9113.616941] Lustre: Skipped 1 previous similar message [ 9118.689919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9118.698319] Lustre: Skipped 3 previous similar messages [ 9123.817157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9123.828537] Lustre: Skipped 2 previous similar messages [ 9125.343364] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9125.491853] Lustre: server umount lustre-MDT0000 complete [ 9128.930890] LustreError: 209538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9128.951703] LustreError: 209538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9130.496818] LustreError: 209519:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314171 with bad export cookie 17952039441558911192 [ 9130.518686] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9130.533435] LustreError: 209519:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9131.039779] Lustre: server umount lustre-MDT0001 complete [ 9144.632084] Lustre: server umount lustre-OST0000 complete [ 9155.457975] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 9169.811103] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9214.146722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9214.424404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9214.471054] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9214.596323] Lustre: lustre-MDT0000: new disk, initializing [ 9214.709321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9214.722498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9220.188359] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9232.807553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9232.985898] Lustre: 214932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9233.051829] Lustre: *** cfs_fail_loc=131e, val=0*** [ 9233.056059] Lustre: Skipped 1 previous similar message [ 9233.060562] LustreError: 215715:0:(llog.c:1696:llog_backup()) MGC192.168.202.102@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 9233.072600] LustreError: 215715:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 9233.077559] Lustre: 215715:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.102@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 9233.125784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9233.135613] Lustre: Skipped 1 previous similar message [ 9233.212877] Lustre: lustre-MDT0001: new disk, initializing [ 9233.311930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9233.359451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9233.373351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9238.529910] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9251.961174] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9260.009259] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9268.233995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9268.594259] Lustre: *** cfs_fail_loc=131e, val=0*** [ 9268.595726] Lustre: Skipped 1 previous similar message [ 9268.597294] LustreError: 216877:0:(llog.c:1696:llog_backup()) MGC192.168.202.102@tcp: failed to backup log lustre-OST0000: rc = -28 [ 9268.601077] LustreError: 216877:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 9268.603658] Lustre: 216877:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.102@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 9268.607183] Lustre: 216877:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 9268.621362] Lustre: lustre-OST0000: new disk, initializing [ 9268.625372] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9268.641985] Lustre: 216900:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9268.804107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9275.967381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9275.977882] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9276.130203] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9277.176624] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9288.143435] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9294.328673] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9294.641463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9299.880561] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9300.168571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9307.106184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9307.119139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9307.127760] Lustre: Skipped 1 previous similar message [ 9307.141522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9308.736635] Lustre: server umount lustre-OST0000 complete [ 9315.304613] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9315.312748] LustreError: Skipped 1 previous similar message [ 9315.321619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9315.336727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9315.347688] Lustre: Skipped 1 previous similar message [ 9317.345123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9319.335687] Lustre: server umount lustre-MDT0000 complete [ 9322.466450] LustreError: 214944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9322.498468] LustreError: 214944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9323.804393] LustreError: 214925:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314364 with bad export cookie 17952039441558911913 [ 9323.811077] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9323.820583] LustreError: 214925:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9324.236115] Lustre: server umount lustre-MDT0001 complete [ 9335.263280] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 18:26:14 (1786314374) [ 9363.511695] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9378.239410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9378.729191] LustreError: 220053:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9378.760531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9383.475947] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9389.025590] LustreError: 220054:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9389.061554] LustreError: 220054:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9394.653153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9400.684212] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9404.447290] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9412.644344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9412.988984] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9412.994965] Lustre: Skipped 1 previous similar message [ 9421.901832] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9433.927172] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9434.194251] Lustre: lustre-OST0001: new disk, initializing [ 9434.202327] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9434.208193] Lustre: 222362:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9441.724351] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9443.960590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9443.972826] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9444.101830] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9453.385564] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9460.779974] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9463.461636] Lustre: DEBUG MARKER: ADD OST3 [ 9469.182702] loop: module loaded [ 9469.292597] loop0: detected capacity change from 0 to 204800000 [ 9469.684468] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9469.705687] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9469.721245] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9477.073174] loop0: detected capacity change from 0 to 204800000 [ 9477.526957] Lustre: DEBUG MARKER: STOP OST3 [ 9485.401760] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9485.798823] Lustre: lustre-OST0003: new disk, initializing [ 9485.804671] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 9485.826074] Lustre: 224418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9486.112759] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 9486.128573] Lustre: Skipped 1 previous similar message [ 9486.931224] Lustre: *** cfs_fail_loc=2109, val=2*** [ 9487.133058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 9487.154820] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 9487.226922] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 9488.994172] LustreError: 224491:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 9489.598052] Lustre: Failing over lustre-OST0003 [ 9489.716944] Lustre: server umount lustre-OST0003 complete [ 9492.373884] LustreError: 221537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9492.411381] LustreError: 221537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9492.452919] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9492.452975] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9495.508674] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9507.300749] Lustre: DEBUG MARKER: START OST3 again [ 9514.644253] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9515.113315] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 9516.301497] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 9516.397844] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 9516.400153] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 9524.186523] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9550.174493] Lustre: Failing over lustre-OST0003 [ 9550.288622] Lustre: server umount lustre-OST0003 complete [ 9550.858137] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9550.885897] LustreError: 222827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9550.890369] Lustre: Skipped 2 previous similar messages [ 9550.935318] LustreError: 222827:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 9571.300306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9571.322835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9571.329937] Lustre: Skipped 2 previous similar messages [ 9571.347456] Lustre: Skipped 5 previous similar messages [ 9577.906736] Lustre: server umount lustre-MDT0000 complete [ 9582.788384] LustreError: 223133:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314623 with bad export cookie 17952039441558912795 [ 9582.804881] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9582.807511] LustreError: 223133:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9587.681126] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9587.693229] Lustre: Skipped 7 previous similar messages [ 9597.407381] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9597.709575] Lustre: server umount lustre-MDT0001 complete [ 9602.753443] Lustre: server umount lustre-OST0000 complete [ 9608.082211] Lustre: server umount lustre-OST0001 complete [ 9616.790632] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 9627.005788] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9672.346098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9672.605328] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9672.625420] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9672.689237] Lustre: lustre-MDT0000: new disk, initializing [ 9672.774256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9672.780559] Lustre: Skipped 1 previous similar message [ 9672.817487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9678.897814] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9690.666183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9690.816928] Lustre: 229987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9690.908756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9690.917347] Lustre: Skipped 1 previous similar message [ 9690.976678] Lustre: lustre-MDT0001: new disk, initializing [ 9691.134796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9691.153973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9696.329918] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9707.561352] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9714.996464] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9724.316640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9724.811226] Lustre: lustre-OST0000: new disk, initializing [ 9724.814772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9724.822705] Lustre: 231955:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9726.267743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9726.288892] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9726.408567] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9732.422352] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9744.393947] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9750.314035] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9750.532683] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9756.430372] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9756.767421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9762.292286] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9762.306628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9762.320217] Lustre: Skipped 3 previous similar messages [ 9762.337292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9762.343407] Lustre: Skipped 2 previous similar messages [ 9764.931363] Lustre: server umount lustre-OST0000 complete [ 9775.996217] Lustre: server umount lustre-MDT0000 complete [ 9777.632666] LustreError: 229994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9777.658355] LustreError: 229994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [ 9780.253187] LustreError: 229980:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786314821 with bad export cookie 17952039441558915091 [ 9780.271478] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9780.272124] LustreError: 229980:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9780.603643] Lustre: server umount lustre-MDT0001 complete [ 9784.556981] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 18:33:43 (1786314823) [ 9804.012122] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [ 9815.664706] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [ 9863.524957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9863.899975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9863.935893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9864.109604] Lustre: lustre-MDT0000: new disk, initializing [ 9864.287518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9870.222609] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9881.911333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9882.032259] Lustre: 236567:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9882.413129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9888.491656] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9901.124995] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9909.476306] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9919.549396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9919.920287] Lustre: 238540:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9921.551268] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9921.932045] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9930.001470] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [ 9941.597672] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9948.064840] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9948.278743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9953.323176] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9953.527284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9957.861670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9957.863781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9957.872837] LustreError: Skipped 2 previous similar messages [ 9957.876766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9957.885261] Lustre: Skipped 4 previous similar messages [ 9957.901232] Lustre: Skipped 3 previous similar messages [ 9970.143570] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9970.371609] Lustre: server umount lustre-OST0000 complete [ 9974.758354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9980.761175] Lustre: server umount lustre-MDT0000 complete [ 9983.473738] LustreError: 237436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9983.504573] LustreError: 237436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9984.962653] LustreError: 238539:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315025 with bad export cookie 17952039441558915973 [ 9984.971832] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9985.362458] Lustre: server umount lustre-MDT0001 complete [ 9995.505921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9996.253226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9996.255943] Lustre: Skipped 5 previous similar messages [10001.422971] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10011.421087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10016.913880] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10025.674796] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10032.157268] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10039.222742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10046.326941] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10055.323965] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10066.404917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10072.355540] Lustre: server umount lustre-OST0000 complete [10084.652838] Lustre: server umount lustre-MDT0000 complete [10089.393580] LustreError: 240741:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315130 with bad export cookie 17952039441558916855 [10089.411345] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10089.415972] LustreError: 240741:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 3 previous similar messages [10089.712442] Lustre: server umount lustre-MDT0001 complete [10198.785086] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [10203.515949] Key type lgssc unregistered [10204.081393] LNet: 244430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10204.097280] LNetError: 244430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10204.122834] LNet: Removed LNI 192.168.202.102@tcp [10205.368871] Key type .llcrypt unregistered [10205.371678] Key type ._llcrypt unregistered [10237.487991] Key type ._llcrypt registered [10237.492190] Key type .llcrypt registered [10237.648280] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [10256.755112] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10257.892629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10258.137480] alg: No test for adler32 (adler32-zlib) [10259.196818] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [10259.517683] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [10261.297309] Key type lgssc registered [10262.411154] Lustre: Echo OBD driver; http://www.lustre.org/ [10305.140662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10305.218207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10306.523168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10306.557720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10306.640831] Lustre: lustre-MDT0000: new disk, initializing [10306.722484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10306.738720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10311.903702] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10322.877646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10323.006371] Lustre: 247784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10323.064428] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10323.068183] Lustre: Skipped 1 previous similar message [10323.141835] Lustre: lustre-MDT0001: new disk, initializing [10323.260311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10323.300317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10323.314936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10328.412626] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10340.788955] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10347.719568] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10356.195300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10356.514613] Lustre: lustre-OST0000: new disk, initializing [10356.520431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10356.528484] Lustre: 249755:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10356.616404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10362.412079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10362.425177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10362.622144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10363.896417] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10376.355528] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10383.278817] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10383.802141] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10390.555180] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10390.708829] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10393.573358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10393.580242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10393.583474] LustreError: Skipped 1 previous similar message [10393.585534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10393.595032] Lustre: Skipped 1 previous similar message [10393.608467] Lustre: Skipped 1 previous similar message [10398.691716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10398.713767] Lustre: Skipped 1 previous similar message [10403.810966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10403.823256] Lustre: Skipped 1 previous similar message [10407.391531] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10407.933728] Lustre: server umount lustre-OST0000 complete [10415.584938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10415.590079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10415.603798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10419.682169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10419.686847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10419.723958] Lustre: Skipped 1 previous similar message [10421.544588] Lustre: server umount lustre-MDT0000 complete [10427.131894] LustreError: 247778:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315467 with bad export cookie 4008025158392522241 [10427.141600] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10427.729872] Lustre: server umount lustre-MDT0001 complete [10453.569575] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10466.331832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10466.947623] LustreError: 252712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10466.960482] LustreError: 252712:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10467.009917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10472.423595] LustreError: 252713:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10472.439906] LustreError: 252713:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10473.199805] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10476.514043] LustreError: 252712:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10481.641099] LustreError: 252713:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10483.097324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10483.468059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10488.101817] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10491.532496] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10498.585497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10498.951601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10506.500820] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10515.284569] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10515.450828] Lustre: lustre-OST0001: new disk, initializing [10515.454600] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10515.460153] Lustre: 255020:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10515.543916] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10521.158683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10521.171842] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10521.300925] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10523.119451] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10536.114897] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10543.839181] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10560.380969] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 18:46:39 (1786315599) [10565.615386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10565.630413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10565.658633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10569.076857] Lustre: server umount lustre-MDT0000 complete [10572.774517] LustreError: 252708:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10572.789220] LustreError: 252708:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10573.335359] LustreError: 253847:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315614 with bad export cookie 4008025158392523123 [10573.340205] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10573.351847] LustreError: 253847:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 1 previous similar message [10573.950712] Lustre: server umount lustre-MDT0001 complete [10588.703534] Lustre: server umount lustre-OST0000 complete [10602.631425] Lustre: server umount lustre-OST0001 complete [10610.385545] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [10624.549418] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [10664.807569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10665.179582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10665.225932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10665.344282] Lustre: lustre-MDT0000: new disk, initializing [10665.497144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10665.530115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10670.871366] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10683.556440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10683.708633] Lustre: 258985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10683.761310] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10683.765044] Lustre: Skipped 1 previous similar message [10683.861270] Lustre: lustre-MDT0001: new disk, initializing [10684.037911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10684.054329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10689.039096] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10699.517703] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10706.150300] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10713.987460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10714.310797] Lustre: lustre-OST0000: new disk, initializing [10714.314656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10714.327267] Lustre: 260957:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10715.792851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10715.802300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10715.949851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10721.676625] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10732.624266] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10737.680581] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10737.959550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10743.537762] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10743.809822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10746.848801] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10746.862579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10746.870257] LustreError: Skipped 1 previous similar message [10746.876172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10746.896346] Lustre: Skipped 4 previous similar messages [10746.898685] Lustre: Skipped 4 previous similar messages [10760.159208] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10760.352494] Lustre: server umount lustre-OST0000 complete [10765.796882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10765.808676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10771.284906] Lustre: server umount lustre-MDT0000 complete [10772.457820] LustreError: 258992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10772.473764] LustreError: 258992:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10774.840973] LustreError: 258979:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786315815 with bad export cookie 4008025158392525258 [10774.853325] LustreError: 258979:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [10774.854594] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10775.234605] Lustre: server umount lustre-MDT0001 complete [10784.801280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10785.304960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10785.312314] Lustre: Skipped 2 previous similar messages [10790.374510] LustreError: 263176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10790.388810] LustreError: 263176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10790.928792] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10801.460676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10807.700215] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10816.706837] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10822.553684] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10831.041488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10838.019398] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [10846.531443] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10994.657767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10994.668179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10994.671058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10994.671067] Lustre: Skipped 8 previous similar messages [10994.683757] Lustre: Skipped 2 previous similar messages [10999.786948] LustreError: 264998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10999.804646] LustreError: 264998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10999.949751] Lustre: server umount lustre-OST0000 complete [11018.207215] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11018.411570] Lustre: server umount lustre-MDT0000 complete [11021.926938] LustreError: 263156:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316062 with bad export cookie 4008025158392526140 [11021.940376] LustreError: 263156:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 1 previous similar message [11021.949701] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11022.249659] Lustre: server umount lustre-MDT0001 complete [11115.456994] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [11119.165395] Key type lgssc unregistered [11119.518910] LNet: 266788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11119.534665] LNetError: 266788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11119.576681] LNet: Removed LNI 192.168.202.102@tcp [11120.573289] Key type .llcrypt unregistered [11120.575469] Key type ._llcrypt unregistered [11149.166874] Key type ._llcrypt registered [11149.169928] Key type .llcrypt registered [11149.308248] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [11165.613532] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11166.520294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11166.670166] alg: No test for adler32 (adler32-zlib) [11167.677739] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [11167.912931] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [11169.583405] Key type lgssc registered [11170.817375] Lustre: Echo OBD driver; http://www.lustre.org/ [11224.210057] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11237.652367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11237.693487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11239.028701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11239.065440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11239.150462] Lustre: lustre-MDT0000: new disk, initializing [11239.258276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11239.276493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11245.173116] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11255.078878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11255.221273] Lustre: 270883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11255.264234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11255.266920] Lustre: Skipped 1 previous similar message [11255.340327] Lustre: lustre-MDT0001: new disk, initializing [11255.433608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11255.474991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11255.495139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11259.841848] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11265.297151] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11272.186439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11272.474575] Lustre: lustre-OST0000: new disk, initializing [11272.478958] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11272.488683] Lustre: 272514:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11272.609671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11274.742138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11274.750959] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11274.853949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11278.805448] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11289.962606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11290.246100] Lustre: lustre-OST0001: new disk, initializing [11290.248777] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11290.253969] Lustre: 273386:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11290.328061] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11297.028759] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11300.051173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11300.065743] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11300.196174] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11309.054244] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11317.151470] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11331.045563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11331.049361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11331.068270] Lustre: Skipped 3 previous similar messages [11331.082952] Lustre: Skipped 2 previous similar messages [11336.195928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11336.205366] Lustre: Skipped 4 previous similar messages [11341.291920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11341.300043] Lustre: Skipped 2 previous similar messages [11344.351148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11344.549337] Lustre: server umount lustre-MDT0000 complete [11348.631848] LustreError: 270877:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316389 with bad export cookie 9304113181982881305 [11348.639547] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11348.648702] LustreError: 270877:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [11349.124594] Lustre: server umount lustre-MDT0001 complete [11363.756311] Lustre: server umount lustre-OST0000 complete [11367.905110] Lustre: 268368:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786316392/real 1786316392] req@ffff9038098a1f80 x1873088304199040/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786316408 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11367.923319] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11368.674094] Lustre: server umount lustre-OST0001 complete [11383.622905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11393.835268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11402.024704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [11412.629881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [11434.970573] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11446.187635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11446.205322] Lustre: lustre-MDT0000: reset Object Index mappings [11446.573953] LustreError: 278407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11446.599402] LustreError: 278407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11446.702660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11451.665212] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11451.883452] LustreError: 278408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11457.000346] LustreError: 278407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11460.207932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11460.222245] Lustre: lustre-MDT0001: reset Object Index mappings [11460.564217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11465.308147] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11468.431458] Lustre: 279550:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11475.242484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11482.167596] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11485.958528] LustreError: 279904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11491.305513] LustreError: 279907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11492.272553] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11492.338843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [11492.669351] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11492.681476] Lustre: Skipped 1 previous similar message [11498.016849] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [11500.934207] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11510.580257] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11514.633060] Lustre: 281420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11521.597321] Lustre: server umount lustre-MDT0000 complete [11522.016971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11522.032296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11522.045142] LustreError: 278407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11522.071692] LustreError: 278407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11526.022522] LustreError: 278388:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316566 with bad export cookie 9304113181982888095 [11526.027389] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11526.035940] LustreError: 278388:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [11526.477194] Lustre: server umount lustre-MDT0001 complete [11541.389490] Lustre: server umount lustre-OST0000 complete [11544.287172] Lustre: 268367:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786316569/real 1786316569] req@ffff90380dd7ca80 x1873088304248704/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786316585 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11544.316837] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11544.329393] Lustre: Skipped 3 previous similar messages [11546.402288] Lustre: server umount lustre-OST0001 complete [11570.629965] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 19:03:29 (1786316609) [11589.404217] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [11600.296867] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11641.234244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11641.501881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11641.523211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11641.584125] Lustre: lustre-MDT0000: new disk, initializing [11641.645966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11641.657394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11645.833605] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11659.263134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11659.420937] Lustre: 285676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11659.498652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11659.500881] Lustre: Skipped 1 previous similar message [11659.560241] Lustre: lustre-MDT0001: new disk, initializing [11659.761325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11659.777013] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11665.463276] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11679.099941] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11686.866719] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11694.587903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11694.913381] Lustre: lustre-OST0000: new disk, initializing [11694.916321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11694.921914] Lustre: 287649:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11696.314342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11696.320577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11696.405962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11702.082626] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11712.371156] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11716.591833] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11716.788060] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11721.685792] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11721.888835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11727.330778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11727.336068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11727.367292] Lustre: Skipped 1 previous similar message [11727.392948] Lustre: Skipped 2 previous similar messages [11729.997302] Lustre: server umount lustre-OST0000 complete [11736.545579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11736.551353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11736.566320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11740.964360] Lustre: server umount lustre-MDT0000 complete [11742.697771] LustreError: 286546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11742.716805] LustreError: 286546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11744.764593] LustreError: 285668:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316785 with bad export cookie 9304113181982890188 [11744.776172] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11744.781262] LustreError: 285668:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [11745.145194] Lustre: server umount lustre-MDT0001 complete [11765.650156] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11777.045236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11777.428350] LustreError: 290582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11777.490796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11777.502875] Lustre: Skipped 2 previous similar messages [11782.761940] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11792.078900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11797.446879] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11800.920984] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11808.032849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11814.869086] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11825.144743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11825.516363] Lustre: lustre-OST0001: new disk, initializing [11825.526766] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11825.547347] Lustre: 292894:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11827.552767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11827.567611] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11827.655659] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11833.492182] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11845.853990] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11851.608888] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11857.972578] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11858.022552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11868.652880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11868.654538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11868.663401] Lustre: Skipped 3 previous similar messages [11868.667153] Lustre: Skipped 3 previous similar messages [11872.674342] Lustre: server umount lustre-MDT0000 complete [11873.768462] LustreError: 291779:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11873.787881] LustreError: 291779:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11876.342835] LustreError: 290565:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786316917 with bad export cookie 9304113181982891070 [11876.344702] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11876.350603] LustreError: 290565:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 1 previous similar message [11876.630863] Lustre: server umount lustre-MDT0001 complete [11891.127080] Lustre: server umount lustre-OST0000 complete [11905.099802] Lustre: server umount lustre-OST0001 complete [11921.896140] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [11933.629065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11934.185689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11934.188748] Lustre: Skipped 3 previous similar messages [11939.326634] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11947.768917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11952.129557] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11954.990260] Lustre: 297095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11961.722394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11969.008655] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11978.458041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11978.576230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11983.879611] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11984.800858] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [11991.873363] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11995.566746] Lustre: 298968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12001.154881] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [12001.235851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [12013.183942] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 19:10:51 (1786317051) [12029.935764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12029.938678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12029.939372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12029.954102] Lustre: Skipped 4 previous similar messages [12029.956264] Lustre: Skipped 4 previous similar messages [12035.587693] Lustre: server umount lustre-MDT0000 complete [12039.901203] LustreError: 295934:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317080 with bad export cookie 9304113181982897629 [12039.915906] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12039.926431] LustreError: 295934:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [12040.161833] LustreError: 295948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12040.193056] LustreError: 295948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [12046.329409] Lustre: server umount lustre-MDT0001 complete [12056.322793] Lustre: server umount lustre-OST0000 complete [12060.768590] Lustre: server umount lustre-OST0001 complete [12068.289316] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [12078.758715] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [12128.339240] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [12138.922621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12139.256698] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12139.289602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12139.373507] Lustre: lustre-MDT0000: new disk, initializing [12139.498504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12144.035480] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12155.165105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12155.290575] Lustre: 303210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12155.605326] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12160.603668] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12172.030199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12172.296197] Lustre: 304841:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12173.881532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12179.783751] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12191.909890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12192.017455] Lustre: 305713:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12192.117147] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12192.123855] Lustre: Skipped 6 previous similar messages [12193.465435] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12193.474205] Lustre: Skipped 1 previous similar message [12193.676444] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12198.036402] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12208.652626] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12212.291767] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [12212.300270] Lustre: Skipped 1 previous similar message [12229.784851] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 19:14:28 (1786317268) [12232.688666] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12232.697958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12232.711062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12232.722670] Lustre: Skipped 5 previous similar messages [12232.736733] Lustre: Skipped 12 previous similar messages [12233.954319] Lustre: server umount lustre-MDT0000 complete [12238.219094] LustreError: 305723:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317278 with bad export cookie 9304113181982905266 [12238.229322] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12238.939788] Lustre: server umount lustre-MDT0001 complete [12253.473780] Lustre: server umount lustre-OST0000 complete [12268.567667] Lustre: server umount lustre-OST0001 complete [12279.597219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12286.140388] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12295.190558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12301.466684] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12311.114419] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12318.941363] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12328.113596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12335.734939] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12340.902659] LustreError: 309794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12340.946140] LustreError: 309794:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [12343.637405] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12371.848835] loop1: detected capacity change from 0 to 204800000 [12372.138496] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12372.156372] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12372.168725] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12377.836409] loop1: detected capacity change from 0 to 204800000 [12383.891080] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12384.011947] LustreError: 307951:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs22381: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [12384.022174] LustreError: 307951:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs22381 [12384.026171] LustreError: 307951:0:(mgs_handler.c:517:mgs_target_reg()) fs22381-MDT0001: Error checking target index: rc = -13 [12384.032532] LustreError: fs22381-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12384.044582] LustreError: 311702:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [12384.054342] LustreError: 311702:0:(tgt_mount.c:2129:server_put_super()) no obd fs22381-MDT0001 [12384.057805] LustreError: 311702:0:(tgt_mount.c:116:server_deregister_mount()) fs22381-MDT0001 not registered [12384.096177] Lustre: server umount fs22381-MDT0001 complete [12384.099444] LustreError: 311702:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [12393.976780] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12394.114500] Lustre: Setting parameter fs22381-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs22381-MDT0001 [12394.155652] Lustre: srv-fs22381-MDT0001: No data found on store. Initialize space. [12394.159156] Lustre: Skipped 4 previous similar messages [12394.269440] Lustre: fs22381-MDT0001: new disk, initializing [12394.279554] Lustre: Skipped 3 previous similar messages [12394.455208] Lustre: srv-fs22381-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12394.458066] Lustre: 312169:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22381-MDT0001-mdtlov: can't create new llog: rc = -115 [12394.476805] LustreError: 312169:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22381-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12400.081994] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12407.531338] Lustre: server umount fs22381-MDT0001 complete [12420.079299] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12421.274827] loop1: detected capacity change from 0 to 204800000 [12426.539641] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12426.986068] Lustre: 313737:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22381-MDT0001-mdtlov: can't create new llog: rc = -115 [12426.996027] LustreError: 313737:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22381-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12431.555701] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12435.985241] Lustre: server umount fs22381-MDT0001 complete [12445.675551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12445.683604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12445.693659] Lustre: Skipped 2 previous similar messages [12445.715244] Lustre: Skipped 2 previous similar messages [12451.476947] Lustre: server umount lustre-MDT0000 complete [12461.521430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12461.710620] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12467.210196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [12468.085133] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12477.621726] loop1: detected capacity change from 0 to 204800000 [12484.399241] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12485.106426] Lustre: 316184:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22381-MDT0001-mdtlov: can't create new llog: rc = -115 [12485.132828] LustreError: 316184:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22381-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [12491.191890] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12497.357805] Lustre: server umount fs22381-MDT0001 complete [12525.547705] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12525.553385] LustreError: Skipped 1 previous similar message [12530.526880] Lustre: server umount lustre-OST0000 complete [12540.901699] Lustre: server umount lustre-MDT0000 complete [12544.592393] LustreError: 307942:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317585 with bad export cookie 9304113181982907954 [12544.617331] LustreError: 307942:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [12544.630260] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12544.946884] Lustre: server umount lustre-MDT0001 complete [12557.256036] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12559.712777] Key type lgssc unregistered [12559.940666] LNet: 318532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12559.945863] LNetError: 318532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12559.967793] LNet: Removed LNI 192.168.202.102@tcp [12560.832174] Key type .llcrypt unregistered [12560.835978] Key type ._llcrypt unregistered [12582.019554] Key type ._llcrypt registered [12582.020930] Key type .llcrypt registered [12582.104687] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [12582.962705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12583.023526] alg: No test for adler32 (adler32-zlib) [12584.038329] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [12584.216428] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [12585.867232] Key type lgssc registered [12586.783219] Lustre: Echo OBD driver; http://www.lustre.org/ [12593.975610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12593.994736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12595.406084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12600.752513] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12605.931715] LustreError: 319809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12605.960931] LustreError: 319809:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12610.104182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12610.520387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12615.133359] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12622.631270] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12628.704764] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12636.537105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12636.987660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12641.093100] LustreError: 321640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12643.548467] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12646.397913] LustreError: 321640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12651.494143] LustreError: 321642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12652.581658] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12655.605287] LustreError: 321642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12655.639445] LustreError: 321642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12665.843198] LustreError: 321642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12665.885330] LustreError: 321642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12676.536718] loop1: detected capacity change from 0 to 204800000 [12676.738800] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12676.749849] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12676.758955] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12682.279915] loop1: detected capacity change from 0 to 204800000 [12686.311966] LustreError: 321863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12686.334310] LustreError: 321863:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [12689.486268] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12689.717169] Lustre: fs20147-OST0000: new disk, initializing [12689.720280] Lustre: srv-fs20147-OST0000: No data found on store. Initialize space. [12689.726944] Lustre: 323547:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12689.872391] Lustre: fs20147-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12698.341259] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12708.525437] Lustre: server umount fs20147-OST0000 complete [12722.156557] LustreError: 321643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12722.192182] LustreError: 321643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 20 previous similar messages [12722.201865] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12723.464965] loop1: detected capacity change from 0 to 204800000 [12730.588361] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12730.831154] Lustre: fs20147-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12737.537056] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12737.566657] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12749.489028] loop2: detected capacity change from 0 to 204800000 [12749.867223] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12749.885106] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12749.895840] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12756.162784] loop2: detected capacity change from 0 to 204800000 [12761.346945] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12761.454714] LustreError: fs20147-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12761.463190] LustreError: fs20147-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12761.470691] LustreError: 326589:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [12761.481203] LustreError: 326589:0:(tgt_mount.c:2129:server_put_super()) no obd fs20147-OST0001 [12761.487636] LustreError: 326589:0:(tgt_mount.c:116:server_deregister_mount()) fs20147-OST0001 not registered [12761.533632] Lustre: server umount fs20147-OST0001 complete [12761.536159] LustreError: 326589:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -13 [12773.423276] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12773.561514] Lustre: fs20147-OST0001: new disk, initializing [12773.566438] Lustre: srv-fs20147-OST0001: No data found on store. Initialize space. [12773.574060] Lustre: 327140:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12773.664993] Lustre: fs20147-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12780.634984] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12788.720987] LustreError: 321640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12788.739400] LustreError: 321640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [12789.300225] Lustre: server umount fs20147-OST0000 complete [12798.836412] Lustre: server umount fs20147-OST0001 complete [12810.953638] loop1: detected capacity change from 0 to 204800000 [12817.123097] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12817.368965] Lustre: fs20147-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12824.378353] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12832.683954] loop2: detected capacity change from 0 to 204800000 [12839.415927] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12846.943443] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [12861.497749] Lustre: server umount fs20147-OST0000 complete [12875.764054] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12876.277687] Lustre: server umount fs20147-OST0001 complete [12891.114112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12891.116739] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12891.119763] Lustre: Skipped 1 previous similar message [12891.136661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12896.232931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12896.248800] Lustre: Skipped 2 previous similar messages [12896.562774] Lustre: server umount lustre-OST0000 complete [12901.344986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12901.350828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12901.381426] Lustre: Skipped 1 previous similar message [12901.399658] Lustre: Skipped 1 previous similar message [12906.464951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12906.474207] Lustre: Skipped 1 previous similar message [12907.362738] Lustre: server umount lustre-MDT0000 complete [12912.298956] LustreError: 319790:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786317953 with bad export cookie 6702669879048644739 [12912.307282] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12912.322910] LustreError: 319790:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 2 previous similar messages [12913.070229] Lustre: server umount lustre-MDT0001 complete [12930.129815] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [12934.498482] Key type lgssc unregistered [12934.885971] LNet: 333073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12934.899682] LNetError: 333073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12934.921940] LNet: Removed LNI 192.168.202.102@tcp [12936.153277] Key type .llcrypt unregistered [12936.185726] Key type ._llcrypt unregistered [12971.446255] Key type ._llcrypt registered [12971.447978] Key type .llcrypt registered [12971.535678] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [12987.827486] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [12988.882215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12989.021690] alg: No test for adler32 (adler32-zlib) [12990.070637] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [12990.313418] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [12992.007232] Key type lgssc registered [12993.468872] Lustre: Echo OBD driver; http://www.lustre.org/ [13041.181300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13041.213323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13042.469826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13042.497577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13042.568629] Lustre: lustre-MDT0000: new disk, initializing [13042.637832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13042.653767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13048.317283] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13060.538661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13060.702210] Lustre: 336933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13060.759329] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13060.763041] Lustre: Skipped 1 previous similar message [13060.832810] Lustre: lustre-MDT0001: new disk, initializing [13060.912933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13060.958044] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13060.974248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13065.234297] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13075.005068] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13080.388424] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13086.698519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13087.019616] Lustre: lustre-OST0000: new disk, initializing [13087.025305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13087.031769] Lustre: 338903:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13087.085088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13093.005437] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13096.049715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13096.069249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13096.177270] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13103.353814] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13105.505847] Lustre: Mounted lustre-client - version 2.17.56_59_g4c906c2 [13106.438642] Lustre: Unmounted lustre-client [13115.888580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13115.897046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13115.903736] Lustre: Skipped 1 previous similar message [13115.919416] Lustre: Skipped 1 previous similar message [13121.009602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13121.019467] Lustre: Skipped 1 previous similar message [13126.144786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13126.150166] Lustre: Skipped 1 previous similar message [13130.207326] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13130.592778] Lustre: server umount lustre-OST0000 complete [13136.359110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13136.380752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13141.474043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13141.485417] Lustre: Skipped 2 previous similar messages [13142.356655] Lustre: server umount lustre-MDT0000 complete [13147.192853] LustreError: 337727:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318187 with bad export cookie 17951091694459405631 [13147.194945] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13147.200572] LustreError: 337727:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [13147.875547] Lustre: server umount lustre-MDT0001 complete [13159.393558] LNet: 1 local NIs in recovery (showing 1): 192.168.202.102@tcp [13167.623824] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [13171.123613] Key type lgssc unregistered [13171.533026] LNet: 341092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13171.538982] LNetError: 341092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13236.072513] LNet: Removed LNI 192.168.202.102@tcp [13237.051417] Key type .llcrypt unregistered [13237.055071] Key type ._llcrypt unregistered [13256.520837] Key type ._llcrypt registered [13256.527470] Key type .llcrypt registered [13256.629021] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [13274.832717] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [13276.305685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13276.335079] alg: No test for adler32 (adler32-zlib) [13277.451592] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [13277.799356] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [13279.527302] Key type lgssc registered [13280.796607] Lustre: Echo OBD driver; http://www.lustre.org/ [13315.155611] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 19:32:33 (1786318353) [13323.799634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13323.824917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13325.223220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13325.276665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13325.367863] Lustre: lustre-MDT0000: new disk, initializing [13325.437604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13325.461486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13330.635354] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13341.743803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13341.945347] Lustre: 344314:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13342.031696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13342.036653] Lustre: Skipped 1 previous similar message [13342.137199] Lustre: lustre-MDT0001: new disk, initializing [13342.257083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13342.321186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13342.333704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13347.389958] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13357.528455] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13364.049955] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13370.843944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13371.114513] Lustre: lustre-OST0000: new disk, initializing [13371.118538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13371.127606] Lustre: 346281:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13371.213077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13376.222919] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13379.196791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13379.210192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13379.354170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13387.623199] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13410.271979] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13410.274477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13410.277729] LustreError: Skipped 1 previous similar message [13410.281844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13410.294070] Lustre: Skipped 1 previous similar message [13410.317060] Lustre: Skipped 1 previous similar message [13415.355126] Lustre: server umount lustre-OST0000 complete [13420.513039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13420.516167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13420.550636] Lustre: Skipped 1 previous similar message [13420.564039] Lustre: Skipped 1 previous similar message [13425.650713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13425.662965] Lustre: Skipped 1 previous similar message [13430.753501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13430.770385] Lustre: Skipped 1 previous similar message [13434.339294] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13434.538859] Lustre: server umount lustre-MDT0000 complete [13438.545153] LustreError: 344308:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318479 with bad export cookie 10940703712828959470 [13438.564881] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13438.568013] LustreError: 344308:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 3 previous similar messages [13438.988047] Lustre: server umount lustre-MDT0001 complete [13452.658572] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [13455.387180] Key type lgssc unregistered [13455.772539] LNet: 348195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13455.776972] LNetError: 348195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13456.804615] LNet: Removed LNI 192.168.202.102@tcp [13457.621601] Key type .llcrypt unregistered [13457.625278] Key type ._llcrypt unregistered [13481.363299] Key type ._llcrypt registered [13481.368101] Key type .llcrypt registered [13481.607239] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [13483.104855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13483.287119] alg: No test for adler32 (adler32-zlib) [13484.312542] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [13484.599676] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [13486.303565] Key type lgssc registered [13488.025387] Lustre: Echo OBD driver; http://www.lustre.org/ [13499.750605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13499.775192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13501.466520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13506.860383] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13511.657596] LustreError: 349473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13511.676756] LustreError: 349473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13516.771273] LustreError: 349474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13517.746027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13518.166590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13524.642503] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13535.836904] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13543.840945] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13552.165387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13552.522403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13560.204654] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13569.009585] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13585.874150] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13587.939812] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13595.104585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13595.120816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13595.150146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13596.569634] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [13597.666708] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13597.674855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13597.702015] Lustre: Skipped 2 previous similar messages [13601.640188] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [13606.761555] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.2@tcp (stopping) [13606.772902] Lustre: Skipped 3 previous similar messages [13607.391293] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13607.663690] Lustre: server umount lustre-MDT0000 complete [13607.909565] LustreError: 350201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13607.930475] LustreError: 350201:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13611.825365] LustreError: 349454:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318652 with bad export cookie 163899544485902006 [13611.831220] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13611.844556] LustreError: 349454:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 4 previous similar messages [13611.878868] LustreError: 351917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.2@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13613.027169] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13613.044033] Lustre: Skipped 1 previous similar message [13616.032142] LustreError: 351917:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13616.042126] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13616.053673] LustreError: 351917:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13616.060037] Lustre: Skipped 2 previous similar messages [13626.335148] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13626.700774] Lustre: server umount lustre-MDT0001 complete [13631.230298] Lustre: server umount lustre-OST0000 complete [13643.301605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13648.264568] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13651.300757] Lustre: 353677:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.102@tcp [13653.896367] Lustre: server umount MGS complete [13665.003155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13666.038071] LustreError: 354250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13666.060309] LustreError: 354250:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13666.132391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13672.388587] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13683.418654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13683.949418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13689.264591] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13700.000149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13700.476219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13707.357790] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13715.929312] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13718.242906] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13724.933558] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13727.034665] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13732.842946] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [13732.856102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13732.865268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13732.869051] Lustre: Skipped 4 previous similar messages [13734.370100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13747.167729] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13747.553684] Lustre: server umount lustre-OST0000 complete [13755.874219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13755.890878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13755.908823] Lustre: Skipped 1 previous similar message [13758.571291] Lustre: server umount lustre-MDT0000 complete [13762.021565] LustreError: 354250:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13762.046819] LustreError: 354250:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [13763.236365] LustreError: 354231:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786318804 with bad export cookie 163899544485902846 [13763.243609] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13763.246976] LustreError: 354231:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 1 previous similar message [13763.804315] Lustre: server umount lustre-MDT0001 complete [13776.026688] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [13778.794215] Key type lgssc unregistered [13778.997724] LNet: 357693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13779.001711] LNetError: 357693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13779.017051] LNet: Removed LNI 192.168.202.102@tcp [13780.240113] Key type .llcrypt unregistered [13780.242416] Key type ._llcrypt unregistered [13805.450773] Key type ._llcrypt registered [13805.453950] Key type .llcrypt registered [13805.703591] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [13824.591950] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [13825.908614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13826.172944] alg: No test for adler32 (adler32-zlib) [13827.334413] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [13827.526068] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [13829.159507] Key type lgssc registered [13830.323832] Lustre: Echo OBD driver; http://www.lustre.org/ [13873.709660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13873.753292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13875.173156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13875.222705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13875.299672] Lustre: lustre-MDT0000: new disk, initializing [13875.421501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13875.437428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13880.445619] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13893.590851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13893.768414] Lustre: 360859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13893.877509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13893.892667] Lustre: Skipped 1 previous similar message [13894.025475] Lustre: lustre-MDT0001: new disk, initializing [13894.139273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13894.186797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13894.201946] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13899.807510] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13912.234895] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13919.083593] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13926.574319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13926.894727] Lustre: lustre-OST0000: new disk, initializing [13926.899443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13926.915452] Lustre: 362831:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13927.060288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13934.829379] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [13935.745357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13935.759858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13935.866607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13946.434344] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13952.248069] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13952.472797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13957.779703] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13957.998706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13961.701970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13961.708336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13961.710073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13961.717275] LustreError: Skipped 1 previous similar message [13961.746801] Lustre: Skipped 1 previous similar message [13966.594038] Lustre: server umount lustre-OST0000 complete [13976.035084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13976.049690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13976.107583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13976.127543] Lustre: Skipped 1 previous similar message [13977.057368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13978.402758] Lustre: server umount lustre-MDT0000 complete [13982.125210] LustreError: 360852:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786319022 with bad export cookie 17549587261263001692 [13982.140814] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13982.146679] LustreError: 360852:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 1 previous similar message [13982.763336] Lustre: server umount lustre-MDT0001 complete [13994.122870] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 19:43:52 (1786319032) [14012.664533] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [14025.265725] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [14071.040290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14071.350779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14071.385644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14071.484832] Lustre: lustre-MDT0000: new disk, initializing [14071.593581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14071.625538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14076.371794] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14086.142571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14086.268639] Lustre: 367593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14086.305809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14086.310187] Lustre: Skipped 1 previous similar message [14086.359420] Lustre: lustre-MDT0001: new disk, initializing [14086.425626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14086.447903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14086.457790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14091.045418] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14100.831073] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14106.647623] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14113.727743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14113.981602] Lustre: lustre-OST0000: new disk, initializing [14113.985838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14113.991837] Lustre: 369559:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14114.074732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14120.785758] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [14123.106169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14123.127309] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14123.254556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14130.830934] Lustre: DEBUG MARKER: oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14169.569602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14169.579634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14169.596167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14169.602522] Lustre: Skipped 1 previous similar message [14169.629932] Lustre: Skipped 3 previous similar messages [14172.548562] Lustre: server umount lustre-OST0000 complete [14178.784956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14178.812201] LustreError: Skipped 1 previous similar message [14178.814330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14178.821668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14183.088484] Lustre: server umount lustre-MDT0000 complete [14187.236050] LustreError: 367584:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786319228 with bad export cookie 17549587261263002574 [14187.239724] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14187.253394] LustreError: 367584:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 5 previous similar messages [14187.601817] Lustre: server umount lustre-MDT0001 complete [14200.732870] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14203.867656] Key type lgssc unregistered [14204.214224] LNet: 371739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14204.244048] LNetError: 371739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14204.281234] LNet: Removed LNI 192.168.202.102@tcp [14205.272062] Key type .llcrypt unregistered [14205.282117] Key type ._llcrypt unregistered [14308.352554] Key type ._llcrypt registered [14308.355262] Key type .llcrypt registered [14308.529166] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14308.862528] Key type .llcrypt unregistered [14308.864457] Key type ._llcrypt unregistered [14392.345692] Key type ._llcrypt registered [14392.350395] Key type .llcrypt registered [14392.492458] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14392.773404] Key type .llcrypt unregistered [14392.775574] Key type ._llcrypt unregistered [14427.268421] Key type ._llcrypt registered [14427.272314] Key type .llcrypt registered [14427.420871] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14427.706461] Key type .llcrypt unregistered [14427.709348] Key type ._llcrypt unregistered [14462.271308] Key type ._llcrypt registered [14462.276665] Key type .llcrypt registered [14462.383638] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14462.706581] Key type .llcrypt unregistered [14462.708308] Key type ._llcrypt unregistered [14545.111286] Key type ._llcrypt registered [14545.112886] Key type .llcrypt registered [14545.200191] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14545.564837] Key type .llcrypt unregistered [14545.566969] Key type ._llcrypt unregistered [14585.340837] Key type ._llcrypt registered [14585.347667] Key type .llcrypt registered [14585.549561] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14585.956692] Key type .llcrypt unregistered [14585.958042] Key type ._llcrypt unregistered [14615.699960] Key type ._llcrypt registered [14615.701879] Key type .llcrypt registered [14615.771831] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14616.001489] Key type .llcrypt unregistered [14616.003232] Key type ._llcrypt unregistered [14648.284760] Key type ._llcrypt registered [14648.286526] Key type .llcrypt registered [14648.376315] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14648.796382] Key type .llcrypt unregistered [14648.802752] Key type ._llcrypt unregistered [14685.861136] Key type ._llcrypt registered [14685.864157] Key type .llcrypt registered [14685.973889] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14686.420292] Key type .llcrypt unregistered [14686.426588] Key type ._llcrypt unregistered [14733.444397] Key type ._llcrypt registered [14733.446511] Key type .llcrypt registered [14733.588430] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14734.222289] Key type .llcrypt unregistered [14734.223908] Key type ._llcrypt unregistered [14809.780733] Key type ._llcrypt registered [14809.786929] Key type .llcrypt registered [14809.971740] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14810.526606] Key type .llcrypt unregistered [14810.532044] Key type ._llcrypt unregistered [14846.552150] Key type ._llcrypt registered [14846.554671] Key type .llcrypt registered [14846.704041] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14847.101488] Key type .llcrypt unregistered [14847.104593] Key type ._llcrypt unregistered [14888.599077] Key type ._llcrypt registered [14888.600793] Key type .llcrypt registered [14888.677166] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [14888.985897] Key type .llcrypt unregistered [14888.992267] Key type ._llcrypt unregistered [14944.927367] Key type ._llcrypt registered [14944.929538] Key type .llcrypt registered [14944.997969] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [14966.294980] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [14967.686673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14967.961068] alg: No test for adler32 (adler32-zlib) [14969.192799] Lustre: Lustre: Build Version: 2.17.56_59_g4c906c2 [14969.376071] LNet: Added LNI 192.168.202.102@tcp [8/256/0/180] [14971.087375] Key type lgssc registered [14972.177602] Lustre: Echo OBD driver; http://www.lustre.org/ [15012.267974] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15012.303308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15013.672431] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [15013.726312] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [15013.815785] Lustre: tst250fs-MDT0000: new disk, initializing [15013.904113] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15013.939870] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15018.939525] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15030.543053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15030.662291] Lustre: 392198:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15030.726808] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [15030.729813] Lustre: Skipped 1 previous similar message [15030.787346] Lustre: tst250fs-MDT0001: new disk, initializing [15030.875515] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15030.918019] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15030.932911] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15035.596853] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15040.735503] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [15047.864458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15048.158184] Lustre: tst250fs-OST0000: new disk, initializing [15048.164033] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [15048.174393] Lustre: 393843:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15048.319180] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15053.510345] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15053.527541] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15053.736013] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15056.030524] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15067.997532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15068.231243] Lustre: tst250fs-OST0001: new disk, initializing [15068.238944] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [15068.246461] Lustre: 394713:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15068.442621] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [15075.511772] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [15075.530879] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [15075.677916] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [15075.713259] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_default_debug -1 all [15082.046244] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [15084.323807] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [15084.332828] Lustre: Skipped 1 previous similar message [15111.656735] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15111.670050] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [15111.688942] Lustre: Skipped 3 previous similar messages [15111.721682] Lustre: Skipped 3 previous similar messages [15115.691456] Lustre: server umount tst250fs-MDT0000 complete [15119.738430] LustreError: 393004:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786320160 with bad export cookie 9262704910617177036 [15119.746176] LustreError: 393004:0:(ldlm_lockd.c:2565:ldlm_cancel_handler()) Skipped 1 previous similar message [15119.748021] LustreError: MGC192.168.202.102@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15120.453983] Lustre: server umount tst250fs-MDT0001 complete [15134.792768] Lustre: server umount tst250fs-OST0000 complete [15137.698593] Lustre: 390425:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786320162/real 1786320162] req@ffff903939925f80 x1873092290877824/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786320178 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15137.733316] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15138.745344] Lustre: server umount tst250fs-OST0001 complete [15158.498837] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [15168.106742] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [15203.224985] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 20:04:01 (1786320241) [15205.602188] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [15225.958821] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing set_hostid [15239.624191] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing load_modules_local [15274.136952] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14896 sec ========= 20:05:13 (1786320313) [15276.070618] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:05:15 (1786320315) === [15279.596895] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:05:19 (1786320319) === [15324.394311] Lustre: DEBUG MARKER: oleg202-server.virtnet: executing unload_modules_local [15327.402109] Key type lgssc unregistered [15327.773856] LNet: 403846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15327.781884] LNetError: 403846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15327.810436] LNet: Removed LNI 192.168.202.102@tcp [15328.950166] Key type .llcrypt unregistered [15328.951945] Key type ._llcrypt unregistered