[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 493518908 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003342] x2apic enabled [ 0.005005] Switched APIC routing to physical x2apic. [ 0.006020] kvm-guest: setup PV IPIs [ 0.009405] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010032] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011020] pid_max: default: 32768 minimum: 301 [ 0.013109] LSM: Security Framework initializing [ 0.014098] Yama: becoming mindful. [ 0.016060] SELinux: Initializing. [ 0.017115] *** VALIDATE selinux *** [ 0.026353] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030819] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031200] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034168] *** VALIDATE tmpfs *** [ 0.036680] *** VALIDATE proc *** [ 0.038162] *** VALIDATE cgroup *** [ 0.039037] *** VALIDATE cgroup2 *** [ 0.040443] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.042071] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.043019] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.044044] Spectre V2 : User space: Vulnerable [ 0.045018] Speculative Store Bypass: Vulnerable [ 0.048576] debug: unmapping init [mem 0xffffffffb4659000-0xffffffffb4660fff] [ 0.051000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.051839] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.052037] ... version: 2 [ 0.053021] ... bit width: 48 [ 0.054034] ... generic registers: 4 [ 0.055022] ... value mask: 0000ffffffffffff [ 0.056027] ... max period: 00007fffffffffff [ 0.057029] ... fixed-purpose events: 3 [ 0.058023] ... event mask: 000000070000000f [ 0.060314] rcu: Hierarchical SRCU implementation. [ 0.062760] smp: Bringing up secondary CPUs ... [ 0.063702] x86: Booting SMP configuration: [ 0.064047] .... node #0, CPUs: #1 #2 #3 [ 0.068139] smp: Brought up 1 node, 4 CPUs [ 0.070026] smpboot: Max logical packages: 1 [ 0.071046] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.222778] node 0 deferred pages initialised in 148ms [ 0.226156] devtmpfs: initialized [ 0.228301] x86/mm: Memory block size: 128MB [ 0.233132] gcov: version magic: 0x41383552 [ 0.236342] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.237100] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.238363] pinctrl core: initialized pinctrl subsystem [ 0.239253] [ 0.239757] ************************************************************* [ 0.240021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.241018] ** ** [ 0.242020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.243021] ** ** [ 0.244018] ** This means that this kernel is built to expose internal ** [ 0.245022] ** IOMMU data structures, which may compromise security on ** [ 0.246020] ** your system. ** [ 0.247022] ** ** [ 0.248017] ** If you see this message and you are not debugging the ** [ 0.249022] ** kernel, report this immediately to your vendor! ** [ 0.250019] ** ** [ 0.251023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.252018] ************************************************************* [ 0.253936] NET: Registered protocol family 16 [ 0.257587] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.260094] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.263128] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.267152] cpuidle: using governor menu [ 0.269552] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.272437] PCI: Using configuration type 1 for base access [ 0.275126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.285155] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.288028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.292130] cryptd: max_cpu_qlen set to 1000 [ 0.295225] ACPI: Added _OSI(Module Device) [ 0.297014] ACPI: Added _OSI(Processor Device) [ 0.298012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.300012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.304091] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.311064] ACPI: Interpreter enabled [ 0.312054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.314015] ACPI: Using IOAPIC for interrupt routing [ 0.316113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.320401] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.331754] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.334044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.338023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.341081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.346450] acpiphp: Slot [2] registered [ 0.348125] acpiphp: Slot [5] registered [ 0.350142] acpiphp: Slot [6] registered [ 0.352147] acpiphp: Slot [7] registered [ 0.353115] acpiphp: Slot [8] registered [ 0.355121] acpiphp: Slot [9] registered [ 0.357126] acpiphp: Slot [10] registered [ 0.359152] acpiphp: Slot [3] registered [ 0.360126] acpiphp: Slot [4] registered [ 0.362112] acpiphp: Slot [11] registered [ 0.364128] acpiphp: Slot [12] registered [ 0.365172] acpiphp: Slot [13] registered [ 0.367104] acpiphp: Slot [14] registered [ 0.369105] acpiphp: Slot [15] registered [ 0.371120] acpiphp: Slot [16] registered [ 0.372124] acpiphp: Slot [17] registered [ 0.374214] acpiphp: Slot [18] registered [ 0.376112] acpiphp: Slot [19] registered [ 0.377106] acpiphp: Slot [20] registered [ 0.379157] acpiphp: Slot [21] registered [ 0.381103] acpiphp: Slot [22] registered [ 0.382114] acpiphp: Slot [23] registered [ 0.384137] acpiphp: Slot [24] registered [ 0.386092] acpiphp: Slot [25] registered [ 0.387128] acpiphp: Slot [26] registered [ 0.389108] acpiphp: Slot [27] registered [ 0.390113] acpiphp: Slot [28] registered [ 0.392155] acpiphp: Slot [29] registered [ 0.394114] acpiphp: Slot [30] registered [ 0.395110] acpiphp: Slot [31] registered [ 0.397072] PCI host bridge to bus 0000:00 [ 0.398019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.401024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.404014] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.407081] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.410028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.413043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.415181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.418159] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.422312] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.432835] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.438016] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.441023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.443024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.446022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.449284] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.452985] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.456050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.459881] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.465014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.478013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.483014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.488531] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.497014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.504015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.527018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.538524] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.546016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.557013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.574018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.583308] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.593021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.599017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.617013] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.626961] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.635015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.643019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.660015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.676561] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.690015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.699012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.725019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.740628] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.750015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.757015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.779018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.790560] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.794399] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.797396] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.799360] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.802236] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.807182] iommu: Default domain type: Passthrough [ 0.810381] SCSI subsystem initialized [ 0.811151] ACPI: bus type USB registered [ 0.813100] usbcore: registered new interface driver usbfs [ 0.815105] usbcore: registered new interface driver hub [ 0.817086] usbcore: registered new device driver usb [ 0.819168] pps_core: LinuxPPS API ver. 1 registered [ 0.821014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.824062] PTP clock support registered [ 0.826122] EDAC MC: Ver: 3.0.0 [ 0.828247] PCI: Using ACPI for IRQ routing [ 0.830050] NetLabel: Initializing [ 0.831012] NetLabel: domain hash size = 128 [ 0.832013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.833110] NetLabel: unlabeled traffic allowed by default [ 0.835141] vgaarb: loaded [ 0.836363] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.838015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.844000] clocksource: Switched to clocksource kvm-clock [ 0.958544] VFS: Disk quotas dquot_6.6.0 [ 0.960074] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.962591] *** VALIDATE ramfs *** [ 0.963814] *** VALIDATE hugetlbfs *** [ 0.965352] pnp: PnP ACPI init [ 0.967608] pnp: PnP ACPI: found 6 devices [ 0.996434] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.999837] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.002365] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.005068] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.008135] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.010878] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.014290] NET: Registered protocol family 2 [ 1.017996] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.023629] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.026837] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.033574] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.037396] TCP: Hash tables configured (established 65536 bind 65536) [ 1.040633] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.044454] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.047659] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.050938] NET: Registered protocol family 1 [ 1.054125] RPC: Registered named UNIX socket transport module. [ 1.056586] RPC: Registered udp transport module. [ 1.058337] RPC: Registered tcp transport module. [ 1.060027] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.062316] NET: Registered protocol family 44 [ 1.064273] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.067722] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.070071] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.072406] PCI: CLS 0 bytes, default 64 [ 1.074980] Unpacking initramfs... [ 2.456837] debug: unmapping init [mem 0xffff89427cc54000-0xffff89427ffbffff] [ 2.461342] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.464080] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.467483] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.000552] Initialise system trusted keyrings [ 3.002488] Key type blacklist registered [ 3.006483] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.015792] zbud: loaded [ 3.018566] *** VALIDATE nfs *** [ 3.019897] *** VALIDATE nfs4 *** [ 3.021693] pstore: using deflate compression [ 3.030958] Platform Keyring initialized [ 3.159893] NET: Registered protocol family 38 [ 3.162056] Key type asymmetric registered [ 3.163810] Asymmetric key parser 'x509' registered [ 3.165890] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.169164] io scheduler mq-deadline registered [ 3.170878] io scheduler kyber registered [ 3.172531] io scheduler bfq registered [ 3.174328] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.177330] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.180454] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.183660] ACPI: Power Button [PWRF] [ 3.190844] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.198229] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.214508] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.227899] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.246383] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.275893] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.310396] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.315429] Non-volatile memory driver v1.3 [ 3.317089] Linux agpgart interface v0.103 [ 3.347762] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.350839] vda: detected capacity change from 0 to 76677120 [ 3.368477] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.371482] vdb: detected capacity change from 0 to 1073741824 [ 3.387643] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.390854] vdc: detected capacity change from 0 to 2621440000 [ 3.405161] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.408373] vdd: detected capacity change from 0 to 2621440000 [ 3.423370] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.426722] vde: detected capacity change from 0 to 4294967296 [ 3.442366] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.445400] vdf: detected capacity change from 0 to 4294967296 [ 3.453308] libphy: Fixed MDIO Bus: probed [ 3.459417] usbcore: registered new interface driver usbserial_generic [ 3.462335] usbserial: USB Serial support registered for generic [ 3.464452] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.468275] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.469869] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.472215] mousedev: PS/2 mouse device common for all mice [ 3.474899] rtc_cmos 00:05: RTC can wake from S4 [ 3.479128] rtc_cmos 00:05: registered as rtc0 [ 3.481334] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.484118] intel_pstate: CPU model not supported [ 3.484388] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.489261] hid: raw HID events driver (C) Jiri Kosina [ 3.489610] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.491563] usbcore: registered new interface driver usbhid [ 3.494926] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.496180] usbhid: USB HID core driver [ 3.500158] drop_monitor: Initializing network drop monitor service [ 3.502636] Initializing XFRM netlink socket [ 3.504609] NET: Registered protocol family 10 [ 3.507618] Segment Routing with IPv6 [ 3.509159] NET: Registered protocol family 17 [ 3.511057] mpls_gso: MPLS GSO support [ 3.516426] RAS: Correctable Errors collector initialized. [ 3.518653] AVX version of gcm_enc/dec engaged. [ 3.520312] AES CTR mode by8 optimization enabled [ 3.595636] sched_clock: Marking stable (3595613462, 0)->(4595714759, -1000101297) [ 3.598932] registered taskstats version 1 [ 3.601057] Loading compiled-in X.509 certificates [ 3.602698] zswap: loaded using pool lzo/zbud [ 3.627055] Key type big_key registered [ 3.638910] Key type encrypted registered [ 3.640296] ima: No TPM chip found, activating TPM-bypass! [ 3.641929] ima: Allocated hash algorithm: sha1 [ 3.643573] ima: No architecture policies found [ 3.644961] evm: Initialising EVM extended attributes: [ 3.646585] evm: security.selinux [ 3.647805] evm: security.ima [ 3.648834] evm: security.capability [ 3.650179] evm: HMAC attrs: 0x1 [ 3.652891] rtc_cmos 00:05: setting system clock to 2026-09-05 12:48:45 UTC (1788612525) [ 3.659089] debug: unmapping init [mem 0xffffffffb5603000-0xffffffffb57fffff] [ 3.662659] debug: unmapping init [mem 0xffffffffb4382000-0xffffffffb4658fff] [ 3.671074] Write protecting the kernel read-only data: 28672k [ 3.674296] debug: unmapping init [mem 0xffffffffb2a03000-0xffffffffb2bfffff] [ 3.677225] debug: unmapping init [mem 0xffffffffb3314000-0xffffffffb33fffff] [ 3.710651] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.727495] systemd[1]: Detected virtualization kvm. [ 3.729270] systemd[1]: Detected architecture x86-64. [ 3.730850] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.756737] systemd[1]: No hostname configured. [ 3.758378] systemd[1]: Set hostname to . [ 3.760323] random: systemd: uninitialized urandom read (16 bytes read) [ 3.762710] systemd[1]: Initializing machine ID from random generator. [ 3.804512] random: ln: uninitialized urandom read (6 bytes read) [ 3.900195] random: systemd: uninitialized urandom read (16 bytes read) [ 3.904252] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.914201] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 3.928438] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.569057] device-mapper: uevent: version 1.0.3 [ 4.571487] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.354335] random: fast init done [ 5.406794] virtio_net virtio0 ens2: renamed from eth0 [ 5.419222] scsi host0: ata_piix [ 5.433205] scsi host1: ata_piix [ 5.435131] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.437807] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.154835] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.155839] random: crng init done [ 10.157465] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.576747] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.711477] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.976969] SELinux: Disabled at runtime. [ 12.035527] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.045904] systemd[1]: Detected virtualization kvm. [ 12.048055] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.532784] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.536292] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.541537] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.547328] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.550989] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.558085] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.568072] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. Starting udev Coldplug all Devic[ 12.680646] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS es... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ 12.986066] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Mounted /mnt. [ OK ] Started udev Kernel Device Manager. [ 13.326864] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.348953] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.435814] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.463462] EDAC sbridge: Ver: 1.1.2 [ 14.929965] Key type dns_resolver registered [ 15.232263] NFS: Registering the id_resolver key type [ 15.234555] Key type id_resolver registered [ 15.236353] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg343-server login: [ 30.508106] hrtimer: interrupt took 12091373 ns [ 61.773871] libcfs: loading out-of-tree module taints kernel. [ 61.825684] Key type ._llcrypt registered [ 61.834172] Key type .llcrypt registered [ 62.005598] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 80.395594] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 82.263352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.279739] alg: No test for adler32 (adler32-zlib) [ 83.780275] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 84.617664] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 86.399256] Key type lgssc registered [ 88.079922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 107.354636] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 150.236343] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 164.718456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 164.752242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 166.046733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 166.114360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 166.255906] Lustre: lustre-MDT0000: new disk, initializing [ 166.347095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 166.362767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 171.652350] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.132655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 186.270747] Lustre: 6505:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 186.329978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 186.334897] Lustre: Skipped 1 previous similar message [ 186.435630] Lustre: lustre-MDT0001: new disk, initializing [ 186.510952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 186.552859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 186.564076] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 191.493372] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.614977] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 207.041304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 207.321240] Lustre: lustre-OST0000: new disk, initializing [ 207.330815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 207.341761] Lustre: 8443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 207.424081] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.241409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 208.256410] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 208.390628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 214.286804] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.340087] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.516716] Lustre: lustre-OST0001: new disk, initializing [ 229.523681] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 229.531940] Lustre: 9517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.610147] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 235.849767] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.116265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 238.122930] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 238.149844] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 248.686912] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 254.475882] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 260.549679] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing check_logdir /tmp/testlogs/ [ 265.515871] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing yml_node [ 270.562855] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 273.266361] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 275.878449] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 277.237638] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 08:53:17 EDT 2026 [ 295.824197] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 297.479904] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 314.855049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.862440] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 314.889738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 319.606893] Lustre: server umount lustre-MDT0000 complete [ 323.165076] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788612845 with bad export cookie 13580896874803902447 [ 323.174934] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 323.183646] LustreError: 6498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 325.088401] LustreError: 6514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.089915] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.105235] LustreError: 6514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 325.114449] Lustre: Skipped 3 previous similar messages [ 325.148405] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 325.157077] Lustre: Skipped 4 previous similar messages [ 328.098516] LustreError: 6514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 328.099902] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 328.121061] LustreError: 6514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 328.140749] Lustre: Skipped 1 previous similar message [ 329.672789] Lustre: server umount lustre-MDT0001 complete [ 333.656704] Lustre: server umount lustre-OST0000 complete [ 337.445590] Lustre: server umount lustre-OST0001 complete [ 343.819743] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 354.383306] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 395.635432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.874254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 395.920579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 395.987235] Lustre: lustre-MDT0000: new disk, initializing [ 396.093792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 396.134877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 400.355978] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 410.499750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.623389] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 410.674464] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 410.679223] Lustre: Skipped 1 previous similar message [ 410.767458] Lustre: lustre-MDT0001: new disk, initializing [ 410.849490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 410.892359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 410.911500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 414.860576] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 423.483281] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 428.470262] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 434.605206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 434.836692] Lustre: lustre-OST0000: new disk, initializing [ 434.840706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 434.846936] Lustre: 17039:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 434.930029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 436.828448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 436.861046] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 436.986578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 440.307677] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 450.040900] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 454.850123] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 455.166972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 460.155521] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 460.360148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 462.821164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.825533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 462.842409] Lustre: Skipped 1 previous similar message [ 462.844065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.865117] Lustre: Skipped 1 previous similar message [ 467.941814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 467.949996] Lustre: Skipped 1 previous similar message [ 468.922378] Lustre: server umount lustre-OST0000 complete [ 477.663987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 477.672060] LustreError: Skipped 1 previous similar message [ 477.679603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.691329] Lustre: Skipped 1 previous similar message [ 477.703162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 479.768116] Lustre: server umount lustre-MDT0000 complete [ 483.296630] LustreError: 15079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 483.341853] LustreError: 15079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 485.208990] LustreError: 17038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788613007 with bad export cookie 13580896874803903973 [ 485.225918] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 485.241953] LustreError: 17038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 485.774256] Lustre: server umount lustre-MDT0001 complete [ 493.494490] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 08:56:52 (1788613012) [ 514.553300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.109937] LustreError: 19915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 515.249775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 519.980973] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 520.685728] LustreError: 19915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.220504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.426960] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 542.557947] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 549.036431] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 551.421412] Lustre: Failing over lustre-MDT0000 [ 552.015128] Lustre: server umount lustre-MDT0000 complete [ 554.537336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.550338] Lustre: Skipped 1 previous similar message [ 554.556324] LustreError: 19910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.573032] LustreError: 19910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 570.726632] Lustre: 3638:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788613076/real 1788613076] req@ffff8943019a0e00 x1875496252105344/t0(0) o400->MGC192.168.203.143@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788613092 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 570.760985] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 570.788108] LustreError: 19915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.813995] LustreError: 19915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 572.767102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.400016] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 581.404758] Lustre: Skipped 1 previous similar message [ 585.807035] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 586.725540] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 586.733130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 586.777614] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 595.333458] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 599.424891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.430452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 599.451038] Lustre: Skipped 2 previous similar messages [ 599.487903] Lustre: Skipped 3 previous similar messages [ 605.842182] Lustre: server umount lustre-MDT0000 complete [ 609.608878] LustreError: 19896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788613131 with bad export cookie 13580896874803905394 [ 609.621608] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 609.632472] LustreError: 19896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 609.944374] Lustre: server umount lustre-MDT0001 complete [ 617.197824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 617.716100] LustreError: 23253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.739610] LustreError: 23253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 622.005814] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 630.930514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 636.410348] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 644.613198] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 650.884519] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 652.874347] Lustre: Failing over lustre-MDT0000 [ 655.127259] Lustre: server umount lustre-MDT0000 complete [ 656.865199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 656.878597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 673.247984] Lustre: 3638:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788613179/real 1788613179] req@ffff89430193b100 x1875496252135168/t0(0) o400->MGC192.168.203.143@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788613195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 673.317820] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 676.352665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 683.487901] LustreError: 3637:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8941c7782d80 x1875496252137984/t0(0) o250->MGC192.168.203.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 683.538533] LustreError: 23982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 683.570541] LustreError: 23982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 683.919206] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 689.136895] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 689.139663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 689.167452] Lustre: Skipped 1 previous similar message [ 689.210887] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 689.448653] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 700.836618] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 709.600721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 709.605797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 709.629299] Lustre: Skipped 1 previous similar message [ 709.641318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 709.641325] Lustre: Skipped 4 previous similar messages [ 711.441493] Lustre: server umount lustre-MDT0000 complete [ 715.594831] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788613237 with bad export cookie 13580896874803906332 [ 715.606772] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 715.614730] LustreError: 23234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 716.207617] Lustre: server umount lustre-MDT0001 complete [ 726.549504] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:00:46 (1788613246) [ 744.532848] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 755.486590] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 796.996420] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 808.523342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 808.726750] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 808.746990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 808.820646] Lustre: lustre-MDT0000: new disk, initializing [ 808.887727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 808.896481] Lustre: Skipped 3 previous similar messages [ 808.911805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 813.624073] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 824.717255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 824.853088] Lustre: 29645:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 825.238813] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 830.337829] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 840.584944] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 846.805290] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 853.653322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 854.060791] Lustre: 31614:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 856.068611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 856.148445] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 856.236476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 856.276104] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 860.798687] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 869.965714] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 878.786733] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 896.992167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 896.997699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.007364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 897.035932] Lustre: Skipped 1 previous similar message [ 897.059012] Lustre: Skipped 2 previous similar messages [ 903.058748] Lustre: server umount lustre-OST0000 complete [ 913.764919] Lustre: server umount lustre-MDT0000 complete [ 917.472588] LustreError: 29657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 917.508663] LustreError: 29657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 917.863289] LustreError: 31613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788613439 with bad export cookie 13580896874803906738 [ 917.865191] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 917.876784] LustreError: 31613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 918.325191] Lustre: server umount lustre-MDT0001 complete [ 930.680718] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 934.389434] Key type lgssc unregistered [ 934.758913] LNet: 33594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.772618] LNetError: 33594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.802330] LNet: Removed LNI 192.168.203.143@tcp [ 935.942156] Key type .llcrypt unregistered [ 935.943987] Key type ._llcrypt unregistered [ 961.370465] Key type ._llcrypt registered [ 961.373889] Key type .llcrypt registered [ 961.491749] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 962.840825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 962.891408] alg: No test for adler32 (adler32-zlib) [ 964.110962] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 964.437585] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 966.129099] Key type lgssc registered [ 967.339247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 975.081603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 975.117834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 976.699047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 981.253572] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 987.113451] LustreError: 35003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 987.153267] LustreError: 35003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 990.583319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.929800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 995.653386] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1004.333670] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1011.116469] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1019.567993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1020.146087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1027.407995] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1034.901188] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1035.541842] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1048.460876] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:06:08 (1788613568) [ 1050.389505] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1052.055079] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1058.807507] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:06:18 (1788613578) [ 1067.766888] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:06:27 (1788613587) [ 1068.694993] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1076.823330] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1076.835131] Lustre: Skipped 2 previous similar messages [ 1078.405153] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:06:38 (1788613598) [ 1081.809515] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1081.816465] Lustre: Skipped 3 previous similar messages [ 1098.397331] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1098.404654] Lustre: Skipped 6 previous similar messages [ 1101.007753] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:07:00 (1788613620) [ 1104.048790] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1107.794218] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1117.163163] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:07:16 (1788613636) [ 1118.957942] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1118.961985] Lustre: Skipped 3 previous similar messages [ 1132.555420] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:07:32 (1788613652) [ 1148.392344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.394217] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1148.396064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1148.406217] Lustre: Skipped 1 previous similar message [ 1153.576966] Lustre: server umount lustre-OST0000 complete [ 1158.627573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1158.634133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1158.643185] Lustre: Skipped 1 previous similar message [ 1158.661780] Lustre: Skipped 2 previous similar messages [ 1163.755951] LustreError: 36017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1163.779156] LustreError: 36017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1163.815621] Lustre: server umount lustre-MDT0000 complete [ 1168.268662] LustreError: 37924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788613690 with bad export cookie 2026342626049013483 [ 1168.276217] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1168.283597] LustreError: 37924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1168.716758] Lustre: server umount lustre-MDT0001 complete [ 1180.424104] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1183.028259] Key type lgssc unregistered [ 1183.331458] LNet: 42855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.339199] LNetError: 42855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1183.351944] LNet: Removed LNI 192.168.203.143@tcp [ 1184.071390] Key type .llcrypt unregistered [ 1184.074095] Key type ._llcrypt unregistered [ 1201.320312] Key type ._llcrypt registered [ 1201.327404] Key type .llcrypt registered [ 1201.437611] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 1216.655812] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1217.720138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1217.897843] alg: No test for adler32 (adler32-zlib) [ 1218.981886] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 1219.226742] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1220.903196] Key type lgssc registered [ 1221.959361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1255.580974] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1255.601297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1256.843779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1256.884718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1256.971798] Lustre: lustre-MDT0000: new disk, initializing [ 1257.061831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1257.075795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1261.142815] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1270.470677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1270.591853] Lustre: 46007:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1270.645616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1270.649991] Lustre: Skipped 1 previous similar message [ 1270.753604] Lustre: lustre-MDT0001: new disk, initializing [ 1270.836496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1270.878739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1270.898772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1275.157107] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1284.681329] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1291.190673] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1298.966964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1299.221308] Lustre: lustre-OST0000: new disk, initializing [ 1299.227514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1299.235830] Lustre: 47973:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1299.328219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1305.916802] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1309.841149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1309.851579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1310.012938] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1316.148879] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1321.250159] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1321.425519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1326.871719] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1327.104881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1330.662331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1330.662465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1330.668146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1330.670925] LustreError: Skipped 1 previous similar message [ 1330.730365] Lustre: Skipped 1 previous similar message [ 1335.438724] Lustre: server umount lustre-OST0000 complete [ 1340.897036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.902431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1340.906843] Lustre: Skipped 1 previous similar message [ 1340.908953] Lustre: Skipped 1 previous similar message [ 1346.104856] Lustre: server umount lustre-MDT0000 complete [ 1350.947697] LustreError: 45997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788613872 with bad export cookie 7918280306608312471 [ 1350.971300] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1350.980312] LustreError: 45997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1351.606677] Lustre: server umount lustre-MDT0001 complete [ 1362.329872] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:11:22 (1788613882) [ 1368.054455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.436237] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1368.455516] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1368.523507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1372.222994] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1373.664808] LustreError: 50443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1378.788600] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1379.910898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.327753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1384.471046] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1391.117116] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1396.834728] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1403.647251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1403.980712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1409.535902] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1417.077981] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1422.760612] LustreError: 52971:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1423.607239] Lustre: Modifying parameter general.timeout=1 in log params [ 1425.329339] Lustre: Modifying parameter general.timeout=3 in log params [ 1425.331979] Lustre: Skipped 1 previous similar message [ 1427.968311] Lustre: Modifying parameter general.timeout=6 in log params [ 1427.974986] Lustre: Skipped 2 previous similar messages [ 1432.856318] Lustre: Modifying parameter general.timeout=11 in log params [ 1432.862436] Lustre: Skipped 4 previous similar messages [ 1441.539296] Lustre: Modifying parameter general.timeout=20 in log params [ 1441.542282] Lustre: Skipped 8 previous similar messages [ 1458.194699] Lustre: Modifying parameter general.timeout=39 in log params [ 1458.199860] Lustre: Skipped 18 previous similar messages [ 1490.229783] Lustre: Modifying parameter general.timeout=74 in log params [ 1490.235522] Lustre: Skipped 34 previous similar messages [ 1550.690849] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:14:30 (1788614070) [ 1553.798415] Lustre: 59923:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1554.707817] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1554.722123] Lustre: Skipped 58 previous similar messages [ 1555.840377] Lustre: 60022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1558.135280] Lustre: 60121:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1562.673916] Lustre: 60329:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1562.684864] Lustre: 60329:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1597.813908] Lustre: 61787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1597.822951] Lustre: 61787:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1597.862991] LustreError: 61787:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1602.627245] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:15:22 (1788614122) [ 1618.407933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1618.420224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1618.433120] Lustre: Skipped 1 previous similar message [ 1618.448809] Lustre: Skipped 3 previous similar messages [ 1622.721370] Lustre: server umount lustre-MDT0000 complete [ 1623.526381] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1623.565838] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1626.483714] LustreError: 50423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788614148 with bad export cookie 7918280306608313353 [ 1626.488819] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1626.499871] LustreError: 50423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1626.773432] Lustre: server umount lustre-MDT0001 complete [ 1641.336365] Lustre: server umount lustre-OST0000 complete [ 1670.248425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1670.534369] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1670.786255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1674.848529] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1684.008179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.123666] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1684.163384] Lustre: 64320:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1690.437914] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1699.693168] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1706.292880] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1713.831840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1714.040754] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1714.479169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1714.503151] Lustre: Skipped 1 previous similar message [ 1721.961468] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1730.053461] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1731.894945] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1731.919156] Lustre: Skipped 20 previous similar messages [ 1732.003386] Lustre: 66888:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1740.256937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1740.269669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1740.284277] Lustre: Skipped 1 previous similar message [ 1740.298207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1745.524723] Lustre: server umount lustre-OST0000 complete [ 1751.009700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1751.011619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1751.017874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1751.017882] Lustre: Skipped 1 previous similar message [ 1751.032526] Lustre: Skipped 2 previous similar messages [ 1751.052441] LustreError: Skipped 1 previous similar message [ 1755.806482] Lustre: server umount lustre-MDT0000 complete [ 1756.130728] LustreError: 64332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1756.158840] LustreError: 64332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1759.634248] LustreError: 64311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788614281 with bad export cookie 7918280306608316461 [ 1759.636722] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1759.652989] LustreError: 64311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1760.047757] Lustre: server umount lustre-MDT0001 complete [ 1770.971255] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 1773.712655] Key type lgssc unregistered [ 1774.010110] LNet: 68094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.024897] LNetError: 68094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.043156] LNet: Removed LNI 192.168.203.143@tcp [ 1774.989341] Key type .llcrypt unregistered [ 1774.995443] Key type ._llcrypt unregistered [ 1804.052838] Key type ._llcrypt registered [ 1804.055077] Key type .llcrypt registered [ 1804.224152] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 1805.599151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1805.690936] alg: No test for adler32 (adler32-zlib) [ 1806.907550] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 1807.219084] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 1808.959188] Key type lgssc registered [ 1810.040035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1820.119844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1820.154677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1822.046114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1828.009980] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1831.396932] LustreError: 69504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1831.416747] LustreError: 69504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1836.513675] LustreError: 69505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1836.977484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1837.657297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1843.084569] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1850.765796] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1856.269257] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1863.771113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1864.481537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.519595] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1880.735228] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1892.696127] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1903.586046] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.590054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1903.595459] Lustre: Skipped 1 previous similar message [ 1903.614803] Lustre: Skipped 2 previous similar messages [ 1907.557900] Lustre: server umount lustre-MDT0000 complete [ 1908.710452] LustreError: 69504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1908.732389] LustreError: 69504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1910.853697] LustreError: 69486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788614432 with bad export cookie 7165703624178178320 [ 1910.859436] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1910.870596] LustreError: 69486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1911.161666] Lustre: server umount lustre-MDT0001 complete [ 1925.261959] Lustre: server umount lustre-OST0000 complete [ 1953.623633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1953.939735] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1953.983889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1954.259389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1959.049820] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1969.008377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.151641] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1969.175366] Lustre: 74375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1969.833071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1974.998563] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 1982.556928] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1988.804368] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1997.140341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1997.350074] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1997.893033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.627564] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2012.478761] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2014.325304] Lustre: 76953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2016.185317] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2020.324176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2020.335110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.355572] Lustre: Skipped 1 previous similar message [ 2020.360778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2020.832482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2025.955490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2025.962794] Lustre: Skipped 2 previous similar messages [ 2026.387345] Lustre: server umount lustre-OST0000 complete [ 2031.083455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2031.087819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2031.088938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2031.097056] Lustre: Skipped 2 previous similar messages [ 2036.196452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2036.208020] Lustre: Skipped 2 previous similar messages [ 2044.383147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2044.793355] Lustre: server umount lustre-MDT0000 complete [ 2046.436881] LustreError: 75492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.465497] LustreError: 75492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2048.563448] LustreError: 76957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788614570 with bad export cookie 7165703624178179153 [ 2048.565890] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2048.579422] LustreError: 76957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2048.913872] Lustre: server umount lustre-MDT0001 complete [ 2060.269026] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2062.826312] Key type lgssc unregistered [ 2063.127438] LNet: 78167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2063.136956] LNetError: 78167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2063.172248] LNet: Removed LNI 192.168.203.143@tcp [ 2063.967293] Key type .llcrypt unregistered [ 2063.968562] Key type ._llcrypt unregistered [ 2088.909802] Key type ._llcrypt registered [ 2088.912873] Key type .llcrypt registered [ 2089.043090] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2089.945764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2090.059735] alg: No test for adler32 (adler32-zlib) [ 2091.126698] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 2091.349466] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 2093.000292] Key type lgssc registered [ 2094.027919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2101.492515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2101.540139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2103.159439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.760045] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2112.491424] LustreError: 79575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2112.502309] LustreError: 79575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2116.562943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2117.348685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2122.388218] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2130.682155] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2137.304835] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2144.333469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2145.074581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2151.660427] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2159.898710] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2166.811597] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2167.607417] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2169.412767] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2169.422921] Lustre: Skipped 1 previous similar message [ 2171.516662] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2171.525604] Lustre: Skipped 1 previous similar message [ 2176.503754] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2176.507531] Lustre: Skipped 4 previous similar messages [ 2184.985174] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2184.997476] Lustre: Skipped 7 previous similar messages [ 2201.966157] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2201.971537] Lustre: Skipped 15 previous similar messages [ 2234.252503] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2234.257797] Lustre: Skipped 33 previous similar messages [ 2274.827021] Lustre: 87678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2277.962476] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:26:37 (1788614797) [ 2280.275736] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2283.016344] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:26:42 (1788614802) [ 2291.683415] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2291.700466] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2291.734655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2294.756308] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2294.760447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2294.773347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.820108] Lustre: Skipped 1 previous similar message [ 2296.043248] Lustre: server umount lustre-OST0000 complete [ 2301.410367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2301.425039] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.441825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2304.993212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2304.996982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.527448] Lustre: server umount lustre-MDT0000 complete [ 2310.112259] LustreError: 80307:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.150039] LustreError: 80307:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2312.074460] LustreError: 79556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788614833 with bad export cookie 8627447006224996653 [ 2312.086677] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2312.103106] LustreError: 79556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2312.599618] Lustre: server umount lustre-MDT0001 complete [ 2326.227293] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2329.321764] Key type lgssc unregistered [ 2329.696816] LNet: 88928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2329.703925] LNetError: 88928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2329.727341] LNet: Removed LNI 192.168.203.143@tcp [ 2330.659207] Key type .llcrypt unregistered [ 2330.663462] Key type ._llcrypt unregistered [ 2335.442470] Key type ._llcrypt registered [ 2335.450031] Key type .llcrypt registered [ 2335.597208] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2341.387711] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules [ 2342.509540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2342.835046] alg: No test for adler32 (adler32-zlib) [ 2343.951613] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 2344.194936] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 2345.919433] Key type lgssc registered [ 2347.851844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2349.568951] LustreError: 90134:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2371.315753] LustreError: 90134:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2371.657175] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2381.535736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2381.574925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2383.068731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2388.094760] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2392.548869] LustreError: 91151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.570802] LustreError: 91151:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2397.664788] LustreError: 91150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.435861] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:28:37 (1788614917) [ 2402.786111] LustreError: 91151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.354963] LustreError: 91184:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2403.381915] LustreError: 91184:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2403.389068] LustreError: 91184:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 2403.849323] Lustre: server umount lustre-MDT0000 complete [ 2418.110398] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 2420.950571] Key type lgssc unregistered [ 2421.279579] LNet: 92677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2421.292853] LNetError: 92677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2421.307154] LNet: Removed LNI 192.168.203.143@tcp [ 2422.031804] Key type .llcrypt unregistered [ 2422.035759] Key type ._llcrypt unregistered [ 2442.356531] Key type ._llcrypt registered [ 2442.366076] Key type .llcrypt registered [ 2442.568580] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2443.543536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2443.560837] alg: No test for adler32 (adler32-zlib) [ 2444.793340] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 2445.013652] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 2446.647234] Key type lgssc registered [ 2448.386323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2458.134837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2458.189128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2459.916432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2465.251890] LustreError: 93901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.277466] LustreError: 93901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2465.837761] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2469.349880] LustreError: 93901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2474.465325] LustreError: 93902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2476.486630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2477.367459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2481.703727] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2489.702744] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2496.558055] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2505.070751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2505.920450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2512.781762] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2520.710758] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2582.882189] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:31:42 (1788615102) [ 2584.765861] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2586.864276] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:31:46 (1788615106) [ 2594.792861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2594.807993] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2594.853765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2596.322971] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2596.327929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2596.343548] Lustre: Skipped 1 previous similar message [ 2596.355696] Lustre: Skipped 2 previous similar messages [ 2599.557715] Lustre: server umount lustre-MDT0000 complete [ 2601.443043] LustreError: 94634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.487740] LustreError: 94634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2603.724776] LustreError: 93881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788615125 with bad export cookie 12052615252608489013 [ 2603.737708] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2603.738122] LustreError: 93881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2604.110640] Lustre: server umount lustre-MDT0001 complete [ 2619.345980] Lustre: server umount lustre-OST0000 complete [ 2631.278579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2631.860284] LustreError: 98224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2632.182537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2637.224552] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2641.378821] LustreError: 98225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.404894] LustreError: 98225:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2648.268014] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2649.505921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2654.528941] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2662.951880] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2669.962785] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2685.426741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2685.653895] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2685.662541] LustreError: 98213:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2685.692244] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2685.702647] LustreError: 100312:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2685.722860] LustreError: 100312:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2685.734308] LustreError: 100312:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2685.801435] Lustre: server umount lustre-OST0000 complete [ 2685.803959] LustreError: 100312:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2695.600416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2695.803467] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2695.810996] LustreError: 98211:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2695.850747] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2695.867298] LustreError: 100780:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -98 [ 2695.883975] LustreError: 100780:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2695.896705] LustreError: 100780:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2696.006961] Lustre: server umount lustre-OST0000 complete [ 2696.017393] LustreError: 100780:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2708.157342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2708.376378] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2708.385441] Lustre: Found index 0 for lustre-OST0000, updating log [ 2708.392782] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2708.541734] Lustre: lustre-OST0000: new disk, initializing [ 2708.560290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2708.578805] Lustre: 101315:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2708.887890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.351751] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2720.695949] Lustre: Failing over lustre-OST0000 [ 2721.056499] Lustre: server umount lustre-OST0000 complete [ 2725.856807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2725.866882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2725.896797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2729.452161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2729.455224] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2734.561196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2734.571837] Lustre: Skipped 2 previous similar messages [ 2740.191236] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2740.447519] Lustre: server umount lustre-MDT0000 complete [ 2740.704460] Lustre: 93211:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788615246/real 1788615246] req@000000000904808f x1875498726697216/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788615262 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2740.762091] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.797213] LustreError: 98221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2740.828813] LustreError: 98221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2744.757193] LustreError: 98206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788615266 with bad export cookie 12052615252608491757 [ 2744.768167] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2744.769901] LustreError: 98206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2745.393955] Lustre: server umount lustre-MDT0001 complete [ 2757.038918] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:34:36 (1788615276) [ 2776.469616] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2791.149240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2791.923359] LustreError: 103664:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2792.124304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2798.277387] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2808.907182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.419627] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2819.007726] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2826.256167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2827.333869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2827.344625] Lustre: Skipped 1 previous similar message [ 2827.360778] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2833.449254] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2833.491626] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 2833.499626] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2834.387610] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2845.682044] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2845.836510] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2845.965579] Lustre: lustre-OST0001: new disk, initializing [ 2845.974682] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2845.992429] Lustre: 105983:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2852.954040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2852.970901] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2853.057456] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2853.847855] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2862.439412] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2869.486887] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2876.387154] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2876.397962] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.424421] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2876.427790] Lustre: Skipped 2 previous similar messages [ 2880.685336] Lustre: server umount lustre-MDT0001 complete [ 2882.372403] LustreError: 105465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2882.402171] LustreError: 105465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2892.080551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2892.222550] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2892.230243] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2892.304459] Lustre: 103652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2892.607677] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.43@tcp (not set up) [ 2892.613321] Lustre: Skipped 4 previous similar messages [ 2893.312822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2893.320037] Lustre: Skipped 1 previous similar message [ 2893.433298] LustreError: 107502:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2893.452276] LustreError: 107502:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2893.471109] LustreError: 103656:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2893.608325] LustreError: 107515:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.143@tcp: cfg command failed: rc = -17 [ 2893.623170] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2893.623170] [ 2898.415191] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2899.317880] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 2911.317898] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:37:10 (1788615430) [ 2932.083648] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 2949.519507] Lustre: 109599:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2971.601649] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2974.820526] Lustre: 110733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3065.313200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3065.325323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3065.337860] Lustre: Skipped 3 previous similar messages [ 3065.350332] Lustre: Skipped 3 previous similar messages [ 3070.431953] LustreError: 105465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3070.447748] LustreError: 105465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 3071.010715] Lustre: server umount lustre-MDT0000 complete [ 3074.975710] LustreError: 103644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788615596 with bad export cookie 12052615252608492758 [ 3074.976977] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3074.992413] LustreError: 103644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3075.867884] Lustre: server umount lustre-MDT0001 complete [ 3082.565231] Lustre: server umount lustre-OST0000 complete [ 3087.183220] Lustre: server umount lustre-OST0001 complete [ 3093.179449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3103.429634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3112.957668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3124.263410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3148.709554] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 3159.868478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.886175] Lustre: lustre-MDT0000: reset Object Index mappings [ 3160.514970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3165.281445] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3174.728440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3174.756612] Lustre: lustre-MDT0001: reset Object Index mappings [ 3180.206645] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3183.481716] Lustre: 115827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3190.590418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3199.070514] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3199.607246] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:232 to 0x280000400:257) [ 3206.140774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3208.750297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3214.324967] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3214.342974] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3215.620725] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3224.076473] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3228.218378] Lustre: 117705:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3290.855624] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:43:30 (1788615810) [ 3316.707850] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3316.711234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3316.741773] Lustre: Skipped 6 previous similar messages [ 3321.562531] Lustre: server umount lustre-MDT0000 complete [ 3326.598099] LustreError: 115427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788615848 with bad export cookie 12052615252608539609 [ 3326.622843] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3326.625106] LustreError: 115427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3326.944272] LustreError: 114675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3326.972372] LustreError: 114675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 3333.430434] Lustre: server umount lustre-MDT0001 complete [ 3344.734588] Lustre: server umount lustre-OST0000 complete [ 3355.796326] Lustre: server umount lustre-OST0001 complete [ 3363.214601] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 3375.414150] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 3415.862478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3416.186960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3416.253230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3416.343140] Lustre: lustre-MDT0000: new disk, initializing [ 3416.429466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3421.056684] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3428.790429] Lustre: server umount lustre-MDT0000 complete [ 3440.096445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3440.538494] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3440.569801] Lustre: 121976:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3440.588552] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3440.877116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3440.887623] Lustre: Skipped 4 previous similar messages [ 3446.125452] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3452.205570] Lustre: server umount lustre-MDT0000 complete [ 3461.774828] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:46:21 (1788615981) [ 3463.595877] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3466.011624] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:46:25 (1788615985) [ 3487.117878] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 3497.954942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3503.367432] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3514.164490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3514.295888] Lustre: 124029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3514.409402] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3514.415082] Lustre: Skipped 1 previous similar message [ 3514.546588] Lustre: lustre-MDT0001: new disk, initializing [ 3514.854752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3514.887397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3520.970680] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3526.204264] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3535.014867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3535.318732] Lustre: lustre-OST0000: new disk, initializing [ 3535.327744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3535.337207] Lustre: 125606:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3536.830373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3536.843560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3536.981232] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3542.517405] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3554.901940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3555.125760] Lustre: lustre-OST0001: new disk, initializing [ 3555.133561] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3555.145886] Lustre: 126477:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3557.034388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3557.046475] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3557.185286] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3562.092951] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 3571.809734] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3575.803184] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4007.455321] Lustre: 132901:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4007.468524] Lustre: 132901:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4007.473204] Lustre: 132901:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4007.479412] Lustre: 132901:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4007.485608] Lustre: 132901:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4007.492960] Lustre: 132901:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4303.107611] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:00:22 (1788616822) [ 4318.688970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4318.706652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4318.714042] Lustre: Skipped 3 previous similar messages [ 4318.726234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4318.730041] Lustre: Skipped 7 previous similar messages [ 4324.640162] Lustre: server umount lustre-MDT0000 complete [ 4325.353918] LustreError: 127284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4325.372727] LustreError: 127284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4328.501199] LustreError: 125605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788616850 with bad export cookie 12052615252608609882 [ 4328.514405] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4328.522725] LustreError: 125605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4330.464772] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4330.489700] Lustre: Skipped 3 previous similar messages [ 4335.210795] Lustre: server umount lustre-MDT0001 complete [ 4345.554488] Lustre: server umount lustre-OST0000 complete [ 4355.687100] Lustre: server umount lustre-OST0001 complete [ 4362.843722] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 4374.396716] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 4413.060572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4413.297530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4413.345286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4413.442366] Lustre: lustre-MDT0000: new disk, initializing [ 4413.542353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4413.548337] Lustre: Skipped 4 previous similar messages [ 4413.564706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4417.622956] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4429.519297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4429.694205] Lustre: 135966:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4429.775721] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4429.780603] Lustre: Skipped 1 previous similar message [ 4429.864647] Lustre: lustre-MDT0001: new disk, initializing [ 4429.991020] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4430.001389] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4434.803918] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4445.458487] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4452.067055] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4459.521593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4459.953573] Lustre: lustre-OST0000: new disk, initializing [ 4459.965689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4459.980148] Lustre: 137935:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4461.705187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4461.715136] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4461.829179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4467.825569] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 4477.754835] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4488.313961] Lustre: lustre-MDD0000: changelog on [ 4491.559585] Lustre: lustre-MDD0001: changelog on [ 6499.274366] Lustre: lustre-MDD0001: changelog off [ 6501.314262] Lustre: 140267:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6502.391801] Lustre: lustre-MDD0000: changelog off [ 6510.050661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6510.072987] Lustre: Skipped 1 previous similar message [ 6510.079428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6510.087111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6510.106736] Lustre: Skipped 9 previous similar messages [ 6515.168645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6515.183832] Lustre: Skipped 1 previous similar message [ 6515.914248] Lustre: server umount lustre-OST0000 complete [ 6520.288710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6520.297991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6520.307936] Lustre: Skipped 2 previous similar messages [ 6520.336773] Lustre: Skipped 1 previous similar message [ 6526.401290] Lustre: server umount lustre-MDT0000 complete [ 6529.762800] LustreError: 136763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788619051 with bad export cookie 12052615252609430779 [ 6529.766984] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6529.775076] LustreError: 136763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6530.116684] Lustre: server umount lustre-MDT0001 complete [ 6540.792173] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 6543.660756] Key type lgssc unregistered [ 6543.914991] LNet: 147081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6543.922863] LNetError: 147081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6543.951156] LNet: Removed LNI 192.168.203.143@tcp [ 6544.781154] Key type .llcrypt unregistered [ 6544.785340] Key type ._llcrypt unregistered [ 6566.677386] Key type ._llcrypt registered [ 6566.681453] Key type .llcrypt registered [ 6566.761689] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 6584.156712] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 6585.344326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6585.411389] alg: No test for adler32 (adler32-zlib) [ 6586.493740] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 6586.772138] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 6588.543158] Key type lgssc registered [ 6589.955284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6634.065724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6634.112602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6635.477437] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6635.503449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6635.602302] Lustre: lustre-MDT0000: new disk, initializing [ 6635.673073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6635.686326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6641.434596] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6653.694267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6653.879573] Lustre: 150280:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6653.963835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6653.968915] Lustre: Skipped 1 previous similar message [ 6654.074248] Lustre: lustre-MDT0001: new disk, initializing [ 6654.210290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6654.255808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6654.272208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6659.191339] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6670.935393] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6677.346176] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6685.918634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6686.241519] Lustre: lustre-OST0000: new disk, initializing [ 6686.244875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6686.252228] Lustre: 152248:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6686.335662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6694.433126] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6696.102735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6696.130415] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6696.333755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6710.654687] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6717.734406] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_module obdecho/obdecho [ 6720.323164] LustreError: 153254:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6720.331809] LustreError: 153254:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6722.488520] LustreError: 153350:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6736.365310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6736.380177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6736.414676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6737.380227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6737.396389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6737.398573] Lustre: Skipped 1 previous similar message [ 6737.416796] Lustre: Skipped 2 previous similar messages [ 6739.352597] Lustre: server umount lustre-MDT0000 complete [ 6744.702766] LustreError: 153691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788619266 with bad export cookie 2991362112689018583 [ 6744.710365] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6744.721162] LustreError: 153691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6745.317558] Lustre: server umount lustre-MDT0001 complete [ 6761.178474] Lustre: server umount lustre-OST0000 complete [ 6785.789683] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 6796.859618] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 6839.523286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6839.950471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6839.989473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6840.125489] Lustre: lustre-MDT0000: new disk, initializing [ 6840.205595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6840.227105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6845.881419] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6859.633643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6859.818770] Lustre: 157067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6859.888954] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6859.895577] Lustre: Skipped 1 previous similar message [ 6859.982700] Lustre: lustre-MDT0001: new disk, initializing [ 6860.101552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6860.187454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6860.199573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6865.664820] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6876.687883] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6884.066482] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6891.013042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6891.340424] Lustre: lustre-OST0000: new disk, initializing [ 6891.344648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6891.351970] Lustre: 159038:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6891.419319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6897.594677] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6901.312980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6901.334304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6901.451734] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6906.824318] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6912.981775] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6913.300651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6918.067163] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6918.312832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6922.216167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6922.221994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6922.233962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6922.250112] Lustre: Skipped 1 previous similar message [ 6922.268630] Lustre: Skipped 1 previous similar message [ 6926.682122] Lustre: server umount lustre-OST0000 complete [ 6931.939437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6931.956413] LustreError: Skipped 1 previous similar message [ 6931.971127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6931.993155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6937.042172] Lustre: server umount lustre-MDT0000 complete [ 6937.570378] LustreError: 157080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6937.589858] LustreError: 157080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6941.079098] LustreError: 157865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788619462 with bad export cookie 2991362112689019542 [ 6941.095879] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6941.546110] Lustre: server umount lustre-MDT0001 complete [ 6945.675392] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:44:25 (1788619465) [ 6947.391135] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6949.085960] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 10:44:29 (1788619469) [ 6957.704289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6958.172322] LustreError: 161449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6958.239300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6962.964326] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6963.680598] LustreError: 161450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6968.805514] LustreError: 161449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6973.117461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6978.517278] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 6987.266699] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6993.725464] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7001.137404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7001.523691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7001.534987] Lustre: Skipped 1 previous similar message [ 7009.138222] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7017.638788] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7029.137635] LustreError: 161476:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x240000bd0:0x2:0x0]: rc = -108 [ 7029.152651] Lustre: 161476:0:(llog_cat_server.c:670:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 records in [0x240000bd0:0x2:0x0]: rc = -108 [ 7029.729893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7029.738698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7029.752789] Lustre: Skipped 1 previous similar message [ 7029.772621] LustreError: 161449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7029.821349] Lustre: server umount lustre-MDT0000 complete [ 7034.587020] LustreError: 161430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788619556 with bad export cookie 2991362112689020424 [ 7034.607541] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7034.607788] LustreError: 161430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7035.383760] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7035.390874] Lustre: Skipped 2 previous similar messages [ 7038.788235] LustreError: 161446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7038.821552] LustreError: 161446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7041.177265] Lustre: server umount lustre-MDT0001 complete [ 7048.670309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7064.981548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7065.344908] LustreError: 165294:0:(llog.c:1655:llog_backup()) MGC192.168.203.143@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7065.350457] Lustre: 165294:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.143@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7069.157670] LustreError: 165299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7069.175105] LustreError: 165299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7104.992409] LustreError: 165300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7105.015122] LustreError: 165300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 21 previous similar messages [ 7125.983318] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7125.988900] Lustre: 165294:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.143@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7166.972647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7169.861670] LustreError: 165320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7169.893344] LustreError: 165320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 43 previous similar messages [ 7172.074771] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7173.131570] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7183.399686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7195.632487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7205.865064] LustreError: 148501:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000afe504d3 x1875503069928320/t0(0) o250->MGC192.168.203.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7211.182177] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7211.514465] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7219.998568] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7221.767575] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7227.943420] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7229.375424] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7252.345615] Lustre: server umount lustre-MDT0000 complete [ 7252.464220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7252.464296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7252.485740] Lustre: Skipped 5 previous similar messages [ 7255.981692] LustreError: 164234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788619777 with bad export cookie 2991362112689026913 [ 7255.984578] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7255.992052] LustreError: 164234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7256.311809] Lustre: server umount lustre-MDT0001 complete [ 7270.396415] Lustre: server umount lustre-OST0000 complete [ 7291.470257] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 7300.961065] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 7341.622757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7341.938347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7341.974824] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7342.091839] Lustre: lustre-MDT0000: new disk, initializing [ 7342.179846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7342.187071] Lustre: Skipped 1 previous similar message [ 7342.208766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7347.418634] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7357.590551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7357.651949] Lustre: 170403:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7357.873426] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7363.236431] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7375.278739] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7381.493916] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7388.504655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7388.858358] Lustre: lustre-OST0000: new disk, initializing [ 7388.861963] Lustre: Skipped 1 previous similar message [ 7388.865094] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7388.867679] Lustre: Skipped 2 previous similar messages [ 7388.873185] Lustre: 172372:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7394.867333] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7398.483172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7398.496042] Lustre: Skipped 1 previous similar message [ 7398.502903] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7398.626282] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7404.221218] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7409.134577] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7409.334518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7413.260582] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7413.454533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7419.364844] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7419.373953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7419.377066] LustreError: Skipped 1 previous similar message [ 7419.381444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7419.381454] Lustre: Skipped 2 previous similar messages [ 7419.405563] Lustre: Skipped 1 previous similar message [ 7421.346663] Lustre: server umount lustre-OST0000 complete [ 7439.327465] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7439.631219] Lustre: server umount lustre-MDT0000 complete [ 7439.849307] LustreError: 170409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7439.873428] LustreError: 170409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 35 previous similar messages [ 7443.307287] LustreError: 170394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788619965 with bad export cookie 2991362112689043706 [ 7443.309449] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7443.322926] LustreError: 170394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7443.659453] Lustre: server umount lustre-MDT0001 complete [ 7452.413928] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 10:52:52 (1788619972) [ 7459.003408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7463.649917] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7471.930577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7476.641603] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7484.440979] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7490.743849] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7498.247265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7505.630274] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7513.930982] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7526.913838] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 10:54:06 (1788620046) [ 7533.542022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7533.546437] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7533.553782] Lustre: Skipped 2 previous similar messages [ 7533.576400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7533.588583] Lustre: Skipped 6 previous similar messages [ 7536.923519] Lustre: server umount lustre-OST0000 complete [ 7546.682442] Lustre: server umount lustre-MDT0000 complete [ 7550.568456] LustreError: 174761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620072 with bad export cookie 2991362112689044588 [ 7550.575188] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7550.575838] LustreError: 174761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7551.111140] Lustre: server umount lustre-MDT0001 complete [ 7561.627689] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 7564.327488] Key type lgssc unregistered [ 7564.805703] LNet: 178512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7564.811669] LNetError: 178512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7564.838074] LNet: Removed LNI 192.168.203.143@tcp [ 7565.904261] Key type .llcrypt unregistered [ 7565.909589] Key type ._llcrypt unregistered [ 7584.698578] Key type ._llcrypt registered [ 7584.701216] Key type .llcrypt registered [ 7584.783622] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 7585.952322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7586.031772] alg: No test for adler32 (adler32-zlib) [ 7587.152518] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 7587.538571] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 7589.330649] Key type lgssc registered [ 7590.604788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7599.656892] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7599.696739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7615.327427] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7620.447348] LustreError: 179752:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.143@tcp: failed processing log, type 4: rc = -110 [ 7625.570635] LustreError: 179752:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7625.583512] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7625.589994] LustreError: MGC192.168.203.143@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7625.598399] LustreError: 179752:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7625.604415] LustreError: 179752:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -22 [ 7625.607732] LustreError: 179752:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7625.699102] Lustre: server umount lustre-OST0000 complete [ 7625.701401] LustreError: 179752:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7635.391356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7636.985992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7642.161309] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7646.190560] LustreError: 180251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7646.217856] LustreError: 180251:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7651.297413] LustreError: 180252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7651.690278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7652.155711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7657.129474] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7665.203224] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7670.844555] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7678.972129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7679.091524] LustreError: 182068:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7679.099578] Lustre: 182068:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.143@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7679.242325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7686.119122] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7694.589990] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7698.407154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7698.420170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7698.439045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7701.987800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7701.993042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7702.017825] Lustre: Skipped 1 previous similar message [ 7702.040063] Lustre: Skipped 2 previous similar messages [ 7703.747822] Lustre: server umount lustre-MDT0000 complete [ 7707.105437] LustreError: 180251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7707.126111] LustreError: 180251:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7707.665046] LustreError: 182096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620229 with bad export cookie 8416359978728369094 [ 7707.673235] LustreError: 182096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7707.674719] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7708.235699] Lustre: server umount lustre-MDT0001 complete [ 7722.461495] Lustre: server umount lustre-OST0000 complete [ 7733.403912] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 7743.272249] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 7780.287544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7780.582175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7780.620513] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7780.743834] Lustre: lustre-MDT0000: new disk, initializing [ 7780.852188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7780.869944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7785.882090] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7797.842115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7797.991927] Lustre: 185592:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7798.065442] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7798.072501] Lustre: Skipped 1 previous similar message [ 7798.162157] Lustre: lustre-MDT0001: new disk, initializing [ 7798.290580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7798.331347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7798.358012] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7803.839965] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7814.918797] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7821.510572] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7829.632505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7829.951557] Lustre: lustre-OST0000: new disk, initializing [ 7829.971360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7829.989492] Lustre: 187563:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7830.088627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7835.718665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7835.734579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7835.932655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7838.291256] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7848.729687] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7854.434612] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7854.673922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7859.537282] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7859.769800] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7861.729608] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7861.730289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7861.745543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7861.752796] Lustre: Skipped 1 previous similar message [ 7861.767364] Lustre: Skipped 1 previous similar message [ 7866.863068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7866.867525] Lustre: Skipped 1 previous similar message [ 7867.883534] Lustre: server umount lustre-OST0000 complete [ 7871.968420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7871.982124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7871.984530] Lustre: Skipped 1 previous similar message [ 7872.028971] Lustre: Skipped 1 previous similar message [ 7878.048994] Lustre: server umount lustre-MDT0000 complete [ 7881.865230] LustreError: 185586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620403 with bad export cookie 8416359978728369815 [ 7881.880661] LustreError: 185586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7881.881909] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7882.366418] Lustre: server umount lustre-MDT0001 complete [ 7892.300115] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:00:11 (1788620411) [ 7918.678360] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 7922.467432] Key type lgssc unregistered [ 7922.791316] LNet: 190509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7922.798736] LNetError: 190509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7922.819959] LNet: Removed LNI 192.168.203.143@tcp [ 7923.657716] Key type .llcrypt unregistered [ 7923.673573] Key type ._llcrypt unregistered [ 7943.945426] Key type ._llcrypt registered [ 7943.947517] Key type .llcrypt registered [ 7944.072101] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 7945.312695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7945.348379] alg: No test for adler32 (adler32-zlib) [ 7946.385644] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 7946.635304] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 7948.319244] Key type lgssc registered [ 7949.442780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7957.128329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7957.172725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7958.888674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7963.396957] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7968.229936] LustreError: 191737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7968.248193] LustreError: 191737:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7971.951825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7972.319292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7977.010969] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 7984.709275] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7990.343372] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7998.280401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7998.438520] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7998.441754] LustreError: 193601:0:(llog.c:1696:llog_backup()) MGC192.168.203.143@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7998.450601] Lustre: 193601:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.143@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7998.544200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8004.718719] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8012.145413] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8017.639634] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8017.648568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8017.651684] Lustre: Skipped 2 previous similar messages [ 8017.671629] Lustre: Skipped 2 previous similar messages [ 8021.035871] Lustre: server umount lustre-MDT0000 complete [ 8023.021516] LustreError: 192465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8023.039959] LustreError: 192465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8024.570184] LustreError: 194360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620546 with bad export cookie 8121435616239482469 [ 8024.574252] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8024.593235] LustreError: 194360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8025.088217] LustreError: 192467:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005a0c09e1 x1875504495881216/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8025.366684] Lustre: server umount lustre-MDT0001 complete [ 8039.985485] Lustre: server umount lustre-OST0000 complete [ 8050.161869] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 8061.360469] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 8097.377344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8097.731699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8097.757460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8097.843538] Lustre: lustre-MDT0000: new disk, initializing [ 8097.956832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8097.984501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8102.285495] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8112.612272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8112.758807] Lustre: 197123:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8112.797342] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8112.801331] Lustre: Skipped 1 previous similar message [ 8112.806200] LustreError: 197906:0:(llog.c:1696:llog_backup()) MGC192.168.203.143@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8112.812778] LustreError: 197906:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8112.817035] Lustre: 197906:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.143@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8112.835220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8112.843614] Lustre: Skipped 1 previous similar message [ 8112.961559] Lustre: lustre-MDT0001: new disk, initializing [ 8113.101307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8113.150747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8113.164995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8117.997428] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8127.421215] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8133.826842] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8141.323967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8141.551037] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8141.554312] Lustre: Skipped 1 previous similar message [ 8141.556561] LustreError: 199069:0:(llog.c:1696:llog_backup()) MGC192.168.203.143@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8141.570694] LustreError: 199069:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8141.578733] Lustre: 199069:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.143@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8141.590116] Lustre: 199069:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8141.620968] Lustre: lustre-OST0000: new disk, initializing [ 8141.625839] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8141.640565] Lustre: 199092:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8141.772875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8147.989362] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8150.100864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8150.111333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8150.242489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8156.970313] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8161.498502] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8161.665464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8166.506196] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8166.716351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8170.982208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8170.984879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8170.992447] LustreError: Skipped 1 previous similar message [ 8170.994074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8171.005524] Lustre: Skipped 1 previous similar message [ 8171.021899] Lustre: Skipped 1 previous similar message [ 8174.752935] Lustre: server umount lustre-OST0000 complete [ 8179.681890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8179.693720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8179.708221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8184.701831] Lustre: server umount lustre-MDT0000 complete [ 8186.350081] LustreError: 197136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8188.296888] LustreError: 197115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620710 with bad export cookie 8121435616239483190 [ 8188.302653] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8188.310890] LustreError: 197115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8188.668271] Lustre: server umount lustre-MDT0001 complete [ 8197.881931] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:05:17 (1788620717) [ 8215.599556] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 8225.550808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8225.925497] LustreError: 202220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8225.940635] LustreError: 202220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8225.978298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8230.479496] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8231.408442] LustreError: 202221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8239.738718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8245.526522] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8248.642966] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8255.435042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8262.923743] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8272.248959] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8272.459697] Lustre: lustre-OST0001: new disk, initializing [ 8272.465671] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8272.475900] Lustre: 204528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8272.601043] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8272.604538] Lustre: Skipped 2 previous similar messages [ 8279.661494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8279.680744] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8279.817797] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8279.976023] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8290.858798] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8302.455560] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8304.948957] Lustre: DEBUG MARKER: ADD OST3 [ 8310.622431] loop: module loaded [ 8310.688482] loop0: detected capacity change from 0 to 204800000 [ 8310.940273] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8310.959924] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8310.971081] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8318.621299] loop0: detected capacity change from 0 to 204800000 [ 8318.667568] Lustre: DEBUG MARKER: STOP OST3 [ 8325.171793] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8325.515913] Lustre: lustre-OST0003: new disk, initializing [ 8325.521885] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8325.538384] Lustre: 206576:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8326.803393] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8326.972915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8326.992124] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8327.057923] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8328.777586] Lustre: Failing over lustre-OST0003 [ 8328.867135] LustreError: 206685:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8329.249108] Lustre: server umount lustre-OST0003 complete [ 8332.284026] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8332.305115] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8332.334243] Lustre: Skipped 1 previous similar message [ 8332.351733] LustreError: 205424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8332.393455] LustreError: 205424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8333.646737] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8342.720308] Lustre: DEBUG MARKER: START OST3 again [ 8352.299823] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8352.663783] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8352.678797] Lustre: Skipped 1 previous similar message [ 8352.700894] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8353.098238] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8354.000036] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8354.001327] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8359.808232] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8381.714812] Lustre: Failing over lustre-OST0003 [ 8381.953041] Lustre: server umount lustre-OST0003 complete [ 8383.498788] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8383.510580] Lustre: Skipped 2 previous similar messages [ 8383.512906] LustreError: 205424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8383.544881] LustreError: 205424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 8403.936047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8403.944072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8403.960470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8403.962343] Lustre: Skipped 3 previous similar messages [ 8403.989217] Lustre: Skipped 5 previous similar messages [ 8407.588976] Lustre: server umount lustre-MDT0000 complete [ 8412.926173] LustreError: 202202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620934 with bad export cookie 8121435616239484072 [ 8412.938452] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8412.947829] LustreError: 202202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8414.178420] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8414.191585] Lustre: Skipped 1 previous similar message [ 8417.186393] LustreError: 205113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8417.199813] LustreError: 205113:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 8420.216839] Lustre: server umount lustre-MDT0001 complete [ 8425.423126] Lustre: server umount lustre-OST0000 complete [ 8430.542703] Lustre: server umount lustre-OST0001 complete [ 8437.488232] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 8447.849835] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 8489.537104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8489.857261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8489.905357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8490.024752] Lustre: lustre-MDT0000: new disk, initializing [ 8490.086932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8490.101763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8494.959360] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8506.830493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8507.007524] Lustre: 211989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8507.075774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8507.080914] Lustre: Skipped 1 previous similar message [ 8507.196270] Lustre: lustre-MDT0001: new disk, initializing [ 8507.414760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8507.434361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8512.559574] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8523.570173] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8530.620965] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8538.226986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8538.504695] Lustre: 213957:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8540.434622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8540.458944] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8540.633064] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8545.231764] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8556.241674] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8561.541939] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8561.768359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8567.376359] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8567.629299] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8571.363652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8571.367275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8571.392894] Lustre: Skipped 2 previous similar messages [ 8571.404095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8571.416244] Lustre: Skipped 3 previous similar messages [ 8584.166792] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8584.413200] Lustre: server umount lustre-OST0000 complete [ 8589.280871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8589.284088] LustreError: Skipped 1 previous similar message [ 8589.293357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8589.300130] Lustre: Skipped 4 previous similar messages [ 8595.007547] Lustre: server umount lustre-MDT0000 complete [ 8596.968078] LustreError: 212865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8596.996948] LustreError: 212865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8599.307778] LustreError: 211980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621121 with bad export cookie 8121435616239486361 [ 8599.310536] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8599.327306] LustreError: 211980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8599.635418] Lustre: server umount lustre-MDT0001 complete [ 8603.901993] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:12:03 (1788621123) [ 8621.841474] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 8631.902312] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 8669.696282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8670.056524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8670.090227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8670.095515] Lustre: Skipped 1 previous similar message [ 8670.208356] Lustre: lustre-MDT0000: new disk, initializing [ 8670.212585] Lustre: Skipped 1 previous similar message [ 8670.341661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8675.037520] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8687.877962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8688.071211] Lustre: 218584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8688.558256] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8693.737821] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8706.016138] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8714.269141] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8722.787908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8723.349742] Lustre: 220555:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8725.254747] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8725.489914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8731.940096] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8741.936791] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8746.428904] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8746.598867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8750.872940] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8751.072707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8756.195628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8756.205355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8756.217343] Lustre: Skipped 3 previous similar messages [ 8756.228514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8756.243376] Lustre: Skipped 3 previous similar messages [ 8759.328801] Lustre: server umount lustre-OST0000 complete [ 8777.183209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8777.432338] Lustre: server umount lustre-MDT0000 complete [ 8781.135288] LustreError: 218578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621302 with bad export cookie 8121435616239487250 [ 8781.138894] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8781.145441] LustreError: 218578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8781.549697] Lustre: server umount lustre-MDT0001 complete [ 8791.424482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8792.045926] LustreError: 222774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8792.143652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8792.152473] Lustre: Skipped 5 previous similar messages [ 8796.690863] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8806.250253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8811.803940] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8819.974153] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8826.304520] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8833.544251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8840.182820] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 8848.152917] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8861.672335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8861.685403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8861.686865] LustreError: Skipped 1 previous similar message [ 8861.687641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8861.687645] Lustre: Skipped 7 previous similar messages [ 8861.711427] Lustre: Skipped 4 previous similar messages [ 8864.533436] Lustre: server umount lustre-OST0000 complete [ 8874.366335] Lustre: server umount lustre-MDT0000 complete [ 8878.200178] LustreError: 222754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621400 with bad export cookie 8121435616239488132 [ 8878.208157] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8878.223367] LustreError: 222754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8878.590460] Lustre: server umount lustre-MDT0001 complete [ 8991.119411] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 8995.224151] Key type lgssc unregistered [ 8995.492586] LNet: 226450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8995.497892] LNetError: 226450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8995.511239] LNet: Removed LNI 192.168.203.143@tcp [ 8996.663224] Key type .llcrypt unregistered [ 8996.665169] Key type ._llcrypt unregistered [ 9033.322889] Key type ._llcrypt registered [ 9033.325647] Key type .llcrypt registered [ 9033.455325] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 9052.963672] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 9054.089683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9054.142520] alg: No test for adler32 (adler32-zlib) [ 9055.216301] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 9055.515862] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 9057.175192] Key type lgssc registered [ 9058.527611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9100.902919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9100.952270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9102.268820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9102.312483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9102.441127] Lustre: lustre-MDT0000: new disk, initializing [ 9102.543069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9102.568752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9107.956698] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 9119.265940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9119.399294] Lustre: 229815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9119.438924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9119.442391] Lustre: Skipped 1 previous similar message [ 9119.557267] Lustre: lustre-MDT0001: new disk, initializing [ 9119.723081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9119.778974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9119.798142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9124.883258] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 9134.865794] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9141.982569] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9151.021961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9151.711937] Lustre: lustre-OST0000: new disk, initializing [ 9151.719474] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9151.731247] Lustre: 231784:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9151.969469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9158.743748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9158.754297] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9158.896613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9159.179886] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 9168.967513] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9174.118926] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9174.339487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9179.625465] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9179.928911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9184.739507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9184.746782] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9184.757627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9184.765854] Lustre: Skipped 1 previous similar message [ 9184.777531] Lustre: Skipped 1 previous similar message [ 9188.570917] Lustre: server umount lustre-OST0000 complete [ 9194.988964] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9195.005092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9195.022700] Lustre: Skipped 1 previous similar message [ 9195.051644] Lustre: Skipped 1 previous similar message [ 9199.759953] Lustre: server umount lustre-MDT0000 complete [ 9205.181542] LustreError: 229807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621727 with bad export cookie 3348952506675116825 [ 9205.189645] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9205.194041] LustreError: 229807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9206.025649] Lustre: server umount lustre-MDT0001 complete [ 9228.853518] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 9239.159501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9239.581801] LustreError: 234719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9239.600404] LustreError: 234719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9239.649307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9244.647169] LustreError: 234720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9244.756510] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 9249.762690] LustreError: 234719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9253.423702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9253.738085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9258.206970] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 9261.517661] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9269.035577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9269.462490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9276.054827] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 9285.917645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9286.173492] Lustre: lustre-OST0001: new disk, initializing [ 9286.181554] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9286.196701] Lustre: 237027:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9286.319393] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9291.380991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9291.400194] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9291.597996] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9293.945782] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 9306.015499] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9313.363926] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9327.248453] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:24:06 (1788621846) [ 9330.662241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9330.672167] LustreError: Skipped 1 previous similar message [ 9330.675221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9330.683055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9332.708157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9332.715956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9332.735438] Lustre: Skipped 1 previous similar message [ 9332.739888] Lustre: Skipped 1 previous similar message [ 9335.641825] Lustre: server umount lustre-MDT0000 complete [ 9337.836775] LustreError: 234719:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9337.855103] LustreError: 234719:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9339.494943] LustreError: 234701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621861 with bad export cookie 3348952506675117707 [ 9339.498643] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9339.506617] LustreError: 234701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9339.889332] Lustre: server umount lustre-MDT0001 complete [ 9353.892842] Lustre: server umount lustre-OST0000 complete [ 9368.128561] Lustre: server umount lustre-OST0001 complete [ 9377.196093] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 9393.032055] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 9434.074724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9434.405854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9434.457852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9434.593708] Lustre: lustre-MDT0000: new disk, initializing [ 9434.713988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9434.731790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9440.186953] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 9451.564685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9451.741429] Lustre: 240996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9451.826613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9451.831222] Lustre: Skipped 1 previous similar message [ 9451.928492] Lustre: lustre-MDT0001: new disk, initializing [ 9452.044401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9452.055713] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9456.725586] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 9466.626894] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9472.472203] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9479.516941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9479.790902] Lustre: lustre-OST0000: new disk, initializing [ 9479.814183] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9479.820080] Lustre: 242965:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9481.480505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9481.497983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9481.638353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9486.220959] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 9496.657028] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9501.859963] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9502.079820] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9507.442937] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9507.686644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9512.424056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9512.425076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9512.435065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9512.458333] Lustre: Skipped 2 previous similar messages [ 9512.477728] Lustre: Skipped 3 previous similar messages [ 9516.020827] Lustre: server umount lustre-OST0000 complete [ 9522.658192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9522.677525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9522.683279] Lustre: Skipped 1 previous similar message [ 9526.637579] Lustre: server umount lustre-MDT0000 complete [ 9527.778749] LustreError: 241871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9530.964345] LustreError: 241802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622052 with bad export cookie 3348952506675119842 [ 9530.967797] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9530.977356] LustreError: 241802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9531.406541] Lustre: server umount lustre-MDT0001 complete [ 9541.136069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9541.786145] LustreError: 245180:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9541.801648] LustreError: 245180:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9541.884061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9541.901222] Lustre: Skipped 2 previous similar messages [ 9547.058331] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 9556.208311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9561.822412] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 9571.028707] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9578.212636] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9585.477506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9592.479426] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [ 9599.769095] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9745.897916] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9745.907390] LustreError: Skipped 1 previous similar message [ 9745.913857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9745.929083] Lustre: Skipped 1 previous similar message [ 9745.935537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9749.987978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9752.009869] Lustre: server umount lustre-OST0000 complete [ 9762.642045] Lustre: server umount lustre-MDT0000 complete [ 9765.345962] LustreError: 245180:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9765.378593] LustreError: 245180:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9766.271724] LustreError: 245161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622288 with bad export cookie 3348952506675120724 [ 9766.275689] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9766.282033] LustreError: 245161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9766.674112] Lustre: server umount lustre-MDT0001 complete [ 9869.984381] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [ 9873.407263] Key type lgssc unregistered [ 9873.758141] LNet: 248789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9873.766632] LNetError: 248789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9873.778405] LNet: Removed LNI 192.168.203.143@tcp [ 9874.986179] Key type .llcrypt unregistered [ 9874.988186] Key type ._llcrypt unregistered [ 9903.982642] Key type ._llcrypt registered [ 9903.985727] Key type .llcrypt registered [ 9904.142562] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [ 9920.880670] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 9922.539546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9922.572078] alg: No test for adler32 (adler32-zlib) [ 9923.645694] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [ 9923.940688] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [ 9925.639727] Key type lgssc registered [ 9927.185414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9978.201027] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [ 9989.717857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9989.766222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9990.988466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9991.027911] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9991.106451] Lustre: lustre-MDT0000: new disk, initializing [ 9991.185411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9991.202128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9996.221955] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10007.455505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10007.622564] Lustre: 252860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10007.704348] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10007.708516] Lustre: Skipped 1 previous similar message [10007.791726] Lustre: lustre-MDT0001: new disk, initializing [10007.963241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10007.992503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10008.002678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10013.156462] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10017.683472] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10023.958577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10024.243437] Lustre: lustre-OST0000: new disk, initializing [10024.249475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10024.255221] Lustre: 254490:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10024.336443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10024.688697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10024.707554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10024.817609] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10030.299606] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10040.053453] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10040.227199] Lustre: lustre-OST0001: new disk, initializing [10040.230853] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10040.239831] Lustre: 255363:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10040.340849] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10047.126886] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10048.638660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10048.656137] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10048.803306] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10059.461695] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10064.659964] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10079.722285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10079.725445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10079.736812] Lustre: Skipped 3 previous similar messages [10079.743974] Lustre: Skipped 2 previous similar messages [10083.176608] Lustre: server umount lustre-MDT0000 complete [10087.556112] LustreError: 252852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622609 with bad export cookie 4085543600890745968 [10087.557343] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10087.572419] LustreError: 252852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10088.050761] Lustre: server umount lustre-MDT0001 complete [10102.681637] Lustre: server umount lustre-OST0000 complete [10105.311314] Lustre: 250359:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788622611/real 1788622611] req@ffff8941c2b60e00 x1875506568963968/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788622627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10105.336897] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10107.726317] Lustre: server umount lustre-OST0001 complete [10122.834066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10134.654881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10144.854289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10157.480917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10180.349838] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [10192.824341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10192.841421] Lustre: lustre-MDT0000: reset Object Index mappings [10193.177532] LustreError: 260375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10193.190707] LustreError: 260375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10193.219405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10198.498154] LustreError: 260376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10199.219874] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10203.632255] LustreError: 260375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10208.748884] LustreError: 260376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10211.390912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10211.416370] Lustre: lustre-MDT0001: reset Object Index mappings [10212.279577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10218.307875] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10221.610537] Lustre: 261519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10229.181932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10229.597131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10236.784293] LustreError: 261873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10237.345308] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10238.839852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10247.373759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10252.788025] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10254.436975] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10262.916864] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10266.777791] Lustre: 263390:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10272.653370] Lustre: server umount lustre-MDT0000 complete [10273.256090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10273.276719] LustreError: 260371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10273.277215] Lustre: Skipped 1 previous similar message [10273.310444] LustreError: 260371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10276.263149] LustreError: 260357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622798 with bad export cookie 4085543600890752758 [10276.272384] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10276.279649] LustreError: 260357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10276.623691] Lustre: server umount lustre-MDT0001 complete [10290.998870] Lustre: server umount lustre-OST0000 complete [10294.753274] Lustre: 250362:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788622800/real 1788622800] req@ffff8942fd190380 x1875506569014656/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788622816 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10294.795173] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10294.817209] Lustre: Skipped 2 previous similar messages [10295.066159] Lustre: server umount lustre-OST0001 complete [10315.736645] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:40:35 (1788622835) [10335.405525] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [10349.950790] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [10390.726330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10391.013558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10391.069153] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10391.177520] Lustre: lustre-MDT0000: new disk, initializing [10391.312144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10391.317863] Lustre: Skipped 1 previous similar message [10391.342615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10396.107941] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10407.025951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10407.188950] Lustre: 267657:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10407.252534] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10407.258086] Lustre: Skipped 1 previous similar message [10407.329419] Lustre: lustre-MDT0001: new disk, initializing [10407.491984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10407.509280] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10412.003652] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10421.616473] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10426.833684] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10433.683457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10433.983439] Lustre: lustre-OST0000: new disk, initializing [10433.989685] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10434.004717] Lustre: 269625:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10435.641224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10435.649840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10435.731626] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10440.374978] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10449.918482] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10454.896927] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10455.184977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10459.929910] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10460.204453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10466.797136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10466.806349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10466.807847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10466.838587] Lustre: Skipped 1 previous similar message [10468.630368] Lustre: server umount lustre-OST0000 complete [10473.954358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10473.968814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10477.027205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10477.035046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10477.050080] Lustre: Skipped 2 previous similar messages [10482.149119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10482.160941] Lustre: Skipped 2 previous similar messages [10487.263552] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10487.402172] Lustre: server umount lustre-MDT0000 complete [10491.284795] LustreError: 267651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623013 with bad export cookie 4085543600890754851 [10491.288338] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10491.304486] LustreError: 267651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10491.761332] Lustre: server umount lustre-MDT0001 complete [10512.750853] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [10524.250608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10524.731987] LustreError: 272559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10524.826266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10524.837686] Lustre: Skipped 2 previous similar messages [10529.411960] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10539.169884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10544.961557] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10547.977286] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10553.919631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10560.429169] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10568.983478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10569.271671] Lustre: lustre-OST0001: new disk, initializing [10569.275185] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10569.283321] Lustre: 274870:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10570.441979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10570.460874] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10570.588724] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10575.511464] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10585.036923] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10588.998237] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10594.112911] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10594.225771] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10606.051171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10606.063599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10606.084065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10606.088846] Lustre: Skipped 2 previous similar messages [10609.552260] Lustre: server umount lustre-MDT0000 complete [10612.709861] LustreError: 273514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10612.724829] LustreError: 273514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10612.950564] LustreError: 272540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623134 with bad export cookie 4085543600890755733 [10612.951511] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10612.965829] LustreError: 272540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10613.343700] Lustre: server umount lustre-MDT0001 complete [10627.137901] Lustre: server umount lustre-OST0000 complete [10640.455728] Lustre: server umount lustre-OST0001 complete [10660.202321] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [10672.000221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10672.580140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10672.592423] Lustre: Skipped 3 previous similar messages [10677.736788] LustreError: 278214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10677.754906] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10677.767148] LustreError: 278214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10686.051417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10691.241464] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10694.375700] Lustre: 279354:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10701.999382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10708.218315] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10716.454796] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10717.763936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10717.838054] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10723.134906] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10730.743818] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10734.526262] Lustre: 281233:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10739.912327] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10740.009837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10754.022162] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:47:53 (1788623273) [10773.478701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10773.494614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10773.515975] Lustre: Skipped 3 previous similar messages [10773.523965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10773.535728] Lustre: Skipped 4 previous similar messages [10776.053787] Lustre: server umount lustre-MDT0000 complete [10780.445271] LustreError: 278195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623302 with bad export cookie 4085543600890762131 [10780.452986] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10780.469610] LustreError: 278195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10786.958096] Lustre: server umount lustre-MDT0001 complete [10797.340844] Lustre: server umount lustre-OST0000 complete [10801.303904] Lustre: server umount lustre-OST0001 complete [10808.045230] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [10819.371245] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [10866.001630] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [10876.522479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10876.741527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10876.795507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10876.900260] Lustre: lustre-MDT0000: new disk, initializing [10877.034535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10882.302923] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10894.235144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10894.336447] Lustre: 285762:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10894.593718] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10899.194667] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10910.423409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10910.636330] Lustre: 287395:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10912.202101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10916.817166] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10926.635426] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10926.806328] Lustre: 288266:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10928.483905] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10928.488631] Lustre: Skipped 1 previous similar message [10928.641453] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10933.854952] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [10943.370663] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10946.986453] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10946.989939] Lustre: Skipped 1 previous similar message [10960.404332] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 11:51:20 (1788623480) [10964.452562] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10964.458246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10964.468678] Lustre: Skipped 7 previous similar messages [10964.476682] Lustre: Skipped 10 previous similar messages [10968.733062] Lustre: server umount lustre-MDT0000 complete [10969.574048] LustreError: 285769:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10969.596436] LustreError: 285769:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [10972.330207] LustreError: 285754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623494 with bad export cookie 4085543600890770069 [10972.331066] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10972.340024] LustreError: 285754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10972.647397] Lustre: server umount lustre-MDT0001 complete [10985.388463] Lustre: server umount lustre-OST0000 complete [10999.239808] Lustre: server umount lustre-OST0001 complete [11008.841883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11009.696500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11009.706586] Lustre: Skipped 7 previous similar messages [11014.455763] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11024.053181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11029.482374] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11038.131992] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11045.159613] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11051.651675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11059.219368] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11066.489528] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11086.615267] loop1: detected capacity change from 0 to 204800000 [11086.965150] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11086.991981] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11087.016632] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11092.143759] loop1: detected capacity change from 0 to 204800000 [11098.362462] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11098.483415] LustreError: 290504:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs5700: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11098.502466] LustreError: 290504:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs5700 [11098.511814] LustreError: 290504:0:(mgs_handler.c:517:mgs_target_reg()) fs5700-MDT0001: Error checking target index: rc = -13 [11098.522947] LustreError: fs5700-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11098.531351] LustreError: 294234:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11098.537894] LustreError: 294234:0:(tgt_mount.c:2132:server_put_super()) no obd fs5700-MDT0001 [11098.543363] LustreError: 294234:0:(tgt_mount.c:116:server_deregister_mount()) fs5700-MDT0001 not registered [11098.570407] Lustre: server umount fs5700-MDT0001 complete [11098.579609] LustreError: 294234:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11107.399387] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11107.488379] Lustre: Setting parameter fs5700-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs5700-MDT0001 [11107.559395] Lustre: srv-fs5700-MDT0001: No data found on store. Initialize space. [11107.563748] Lustre: Skipped 4 previous similar messages [11107.637872] Lustre: fs5700-MDT0001: new disk, initializing [11107.640289] Lustre: Skipped 3 previous similar messages [11107.746957] Lustre: srv-fs5700-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11107.759983] Lustre: 294701:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5700-MDT0001-mdtlov: can't create new llog: rc = -115 [11107.781178] LustreError: 294701:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5700-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11112.035984] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11118.298345] Lustre: server umount fs5700-MDT0001 complete [11129.891259] loop1: detected capacity change from 0 to 204800000 [11135.152468] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11135.651496] Lustre: 296270:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5700-MDT0001-mdtlov: can't create new llog: rc = -115 [11135.663194] LustreError: 296270:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5700-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11135.977168] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11140.018230] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11144.262347] Lustre: server umount fs5700-MDT0001 complete [11151.341626] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11152.371967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11152.392137] Lustre: Skipped 1 previous similar message [11152.406107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11158.786843] Lustre: server umount lustre-MDT0000 complete [11166.856298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11166.966167] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11171.817701] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11172.337608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11180.077390] loop1: detected capacity change from 0 to 204800000 [11186.272910] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11186.602908] Lustre: 298716:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5700-MDT0001-mdtlov: can't create new llog: rc = -115 [11186.617831] LustreError: 298716:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5700-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11192.121136] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11197.527061] Lustre: server umount fs5700-MDT0001 complete [11223.009368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11223.025104] LustreError: Skipped 1 previous similar message [11225.715315] Lustre: server umount lustre-OST0000 complete [11235.705866] Lustre: server umount lustre-MDT0000 complete [11238.882198] LustreError: 290515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11238.906690] LustreError: 290515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 106 previous similar messages [11239.149827] LustreError: 290497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623760 with bad export cookie 4085543600890772750 [11239.163196] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11239.169809] LustreError: 290497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11239.622482] Lustre: server umount lustre-MDT0001 complete [11251.340659] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [11253.837683] Key type lgssc unregistered [11254.158796] LNet: 301058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11254.166680] LNetError: 301058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11254.182580] LNet: Removed LNI 192.168.203.143@tcp [11255.013170] Key type .llcrypt unregistered [11255.018079] Key type ._llcrypt unregistered [11274.316176] Key type ._llcrypt registered [11274.322438] Key type .llcrypt registered [11274.443237] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [11275.270035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11275.307524] alg: No test for adler32 (adler32-zlib) [11276.306497] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [11276.514052] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [11278.159164] Key type lgssc registered [11278.973875] Lustre: Echo OBD driver; http://www.lustre.org/ [11285.488981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11285.526614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11286.967808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11290.639127] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11296.227111] LustreError: 302328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11296.242131] LustreError: 302328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11297.750661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11298.112399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11301.810671] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11308.050699] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11313.025924] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11318.926963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11319.246170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11322.408974] LustreError: 304156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11326.365735] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11327.463242] LustreError: 304158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11331.560562] LustreError: 304156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11331.579545] LustreError: 304156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11334.490671] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11336.678665] LustreError: 304159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11336.719373] LustreError: 304159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11346.925128] LustreError: 304158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11346.955330] LustreError: 304158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11353.792490] loop1: detected capacity change from 0 to 204800000 [11354.043956] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11354.070598] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11354.086150] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11359.241990] loop1: detected capacity change from 0 to 204800000 [11365.066037] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11365.303904] Lustre: fs27091-OST0000: new disk, initializing [11365.310637] Lustre: srv-fs27091-OST0000: No data found on store. Initialize space. [11365.325785] Lustre: 306065:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11365.423780] Lustre: fs27091-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11367.397385] LustreError: 304158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11367.428511] LustreError: 304158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11371.160836] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11379.104822] Lustre: server umount fs27091-OST0000 complete [11391.287785] loop1: detected capacity change from 0 to 204800000 [11396.636359] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11396.911734] Lustre: fs27091-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11402.766992] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11403.241925] LustreError: 304156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11403.252113] LustreError: 304156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [11408.359698] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11410.906455] loop2: detected capacity change from 0 to 204800000 [11411.010666] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11411.017533] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11411.023052] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11413.474867] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11415.955244] loop2: detected capacity change from 0 to 204800000 [11421.637192] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11421.727218] LustreError: fs27091-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11421.744061] LustreError: fs27091-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11421.755424] LustreError: 309104:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -13 [11421.770973] LustreError: 309104:0:(tgt_mount.c:2132:server_put_super()) no obd fs27091-OST0001 [11421.778720] LustreError: 309104:0:(tgt_mount.c:116:server_deregister_mount()) fs27091-OST0001 not registered [11421.824947] Lustre: server umount fs27091-OST0001 complete [11421.827928] LustreError: 309104:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11431.296913] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11431.397390] Lustre: fs27091-OST0001: new disk, initializing [11431.400784] Lustre: srv-fs27091-OST0001: No data found on store. Initialize space. [11431.406112] Lustre: 309655:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11431.479227] Lustre: fs27091-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11437.164259] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11444.877688] Lustre: server umount fs27091-OST0000 complete [11453.840824] Lustre: server umount fs27091-OST0001 complete [11466.897804] loop1: detected capacity change from 0 to 204800000 [11469.815197] LustreError: 304829:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11469.851754] LustreError: 304829:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [11472.461865] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11472.645055] Lustre: fs27091-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11478.877751] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11485.920768] loop2: detected capacity change from 0 to 204800000 [11491.151599] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11497.923638] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11506.828550] Lustre: server umount fs27091-OST0000 complete [11515.760655] Lustre: server umount fs27091-OST0001 complete [11531.243699] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11531.244544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11531.254857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11531.260637] Lustre: Skipped 1 previous similar message [11535.308276] Lustre: server umount lustre-OST0000 complete [11538.912689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11538.920057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11538.932512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11538.937723] Lustre: Skipped 1 previous similar message [11541.473287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11541.478600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11544.807629] Lustre: server umount lustre-MDT0000 complete [11548.055701] LustreError: 302309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624069 with bad export cookie 17931613686218324049 [11548.056655] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11548.063341] LustreError: 302309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11548.299753] Lustre: server umount lustre-MDT0001 complete [11560.698717] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [11564.285131] Key type lgssc unregistered [11564.719277] LNet: 315586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11564.730990] LNetError: 315586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11564.752523] LNet: Removed LNI 192.168.203.143@tcp [11565.620149] Key type .llcrypt unregistered [11565.626652] Key type ._llcrypt unregistered [11596.219517] Key type ._llcrypt registered [11596.222581] Key type .llcrypt registered [11596.340270] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [11610.428603] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [11611.238790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11611.267524] alg: No test for adler32 (adler32-zlib) [11612.389188] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [11612.667190] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [11614.351166] Key type lgssc registered [11615.447152] Lustre: Echo OBD driver; http://www.lustre.org/ [11652.070909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11652.107165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11653.448503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11653.484602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11653.591570] Lustre: lustre-MDT0000: new disk, initializing [11653.739126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11653.785358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11658.554482] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11670.875543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11671.048664] Lustre: 319444:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11671.103279] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11671.107824] Lustre: Skipped 1 previous similar message [11671.180832] Lustre: lustre-MDT0001: new disk, initializing [11671.268848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11671.313504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11671.332701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11677.062558] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11687.674888] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11694.095821] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11700.783102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11701.052977] Lustre: lustre-OST0000: new disk, initializing [11701.056897] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11701.062085] Lustre: 321412:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11701.142324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11706.919474] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11708.497021] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11708.508843] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11708.607536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11715.686411] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11717.472708] Lustre: Mounted lustre-client - version 2.17.58_2_ge9089f3 [11718.444251] Lustre: Unmounted lustre-client [11727.842160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11727.850281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11727.864351] Lustre: Skipped 1 previous similar message [11727.882356] Lustre: Skipped 1 previous similar message [11732.922309] Lustre: server umount lustre-OST0000 complete [11738.080300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11738.081278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11738.086735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11738.097863] Lustre: Skipped 1 previous similar message [11738.105458] Lustre: Skipped 1 previous similar message [11742.803843] Lustre: server umount lustre-MDT0000 complete [11746.529386] LustreError: 319438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624268 with bad export cookie 10067597853915969708 [11746.534069] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11746.543847] LustreError: 319438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11746.866155] Lustre: server umount lustre-MDT0001 complete [11756.703854] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [11759.900956] Key type lgssc unregistered [11768.352804] LNet: 1 local NIs in recovery (showing 1): 192.168.203.143@tcp [11768.479606] LNet: 323577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11768.490555] LNetError: 323577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11847.400704] LNet: Removed LNI 192.168.203.143@tcp [11848.708155] Key type .llcrypt unregistered [11848.713189] Key type ._llcrypt unregistered [11871.951108] Key type ._llcrypt registered [11871.952922] Key type .llcrypt registered [11872.176692] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [11887.517428] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [11888.427217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11888.502902] alg: No test for adler32 (adler32-zlib) [11889.625482] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [11889.954110] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [11891.623164] Key type lgssc registered [11892.886693] Lustre: Echo OBD driver; http://www.lustre.org/ [11922.740996] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:07:22 (1788624442) [11928.684090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11928.731977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11930.051662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11930.097963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11930.198620] Lustre: lustre-MDT0000: new disk, initializing [11930.290398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11930.305368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11934.987238] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11946.166946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11946.293377] Lustre: 326797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11946.384564] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11946.388118] Lustre: Skipped 1 previous similar message [11946.466631] Lustre: lustre-MDT0001: new disk, initializing [11946.591447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11946.627068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11946.644731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11951.070811] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11959.888735] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11965.556633] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11972.493975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11972.782729] Lustre: lustre-OST0000: new disk, initializing [11972.787799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11972.798740] Lustre: 328763:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11972.892116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11978.506847] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [11979.811118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11979.822354] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11979.945878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11986.968965] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12000.741578] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12000.772091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12005.344620] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12005.354459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12005.371807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12006.622912] Lustre: server umount lustre-OST0000 complete [12010.978981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12010.980857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12010.997035] Lustre: Skipped 1 previous similar message [12011.014453] Lustre: Skipped 2 previous similar messages [12016.098035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12016.107822] Lustre: Skipped 1 previous similar message [12016.513051] Lustre: server umount lustre-MDT0000 complete [12020.346552] LustreError: 326787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624542 with bad export cookie 4520676500273884999 [12020.350480] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12020.356025] LustreError: 326787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12020.750160] Lustre: server umount lustre-MDT0001 complete [12031.527692] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [12034.066066] Key type lgssc unregistered [12034.380211] LNet: 330669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12034.388278] LNetError: 330669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12034.411530] LNet: Removed LNI 192.168.203.143@tcp [12035.282381] Key type .llcrypt unregistered [12035.288602] Key type ._llcrypt unregistered [12057.711224] Key type ._llcrypt registered [12057.718184] Key type .llcrypt registered [12057.859635] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [12058.739049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12058.789113] alg: No test for adler32 (adler32-zlib) [12059.991908] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [12060.273065] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [12061.975190] Key type lgssc registered [12063.503825] Lustre: Echo OBD driver; http://www.lustre.org/ [12071.820302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12071.846203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12073.292076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12077.685314] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [12083.690599] LustreError: 331952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12083.731147] LustreError: 331952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12087.158332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12087.866050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12093.036522] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [12102.321974] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12108.444343] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12115.888629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12116.227906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12122.865631] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [12130.090938] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12140.642523] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12142.450466] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12148.063517] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.43@tcp (stopping) [12148.194433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12148.210942] Lustre: Skipped 1 previous similar message [12152.656215] Lustre: server umount lustre-MDT0000 complete [12153.181471] LustreError: 331948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12156.699464] LustreError: 331934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624678 with bad export cookie 9744952566856859611 [12156.710940] LustreError: 331934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12156.713758] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12158.280925] LustreError: 331948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12158.310941] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.43@tcp (stopping) [12158.327161] LustreError: 331948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12158.351850] Lustre: Skipped 3 previous similar messages [12158.433485] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12158.449641] Lustre: Skipped 1 previous similar message [12161.451705] LustreError: 333792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12161.453813] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12161.465184] LustreError: 333792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12161.493321] Lustre: Skipped 1 previous similar message [12163.275306] Lustre: server umount lustre-MDT0001 complete [12167.383096] Lustre: server umount lustre-OST0000 complete [12176.744351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12182.109658] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [12184.917848] Lustre: 336134:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.143@tcp [12187.128158] Lustre: server umount MGS complete [12196.926209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12197.602601] LustreError: 336706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12197.730076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12202.714269] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [12208.101700] LustreError: 336706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12208.125787] LustreError: 336706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12212.094811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12212.594980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12217.331253] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [12227.155894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12227.653986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12234.486606] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [12242.535661] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12244.221454] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12250.627333] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12252.255594] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12258.278578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12258.279089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12258.304167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12258.311421] LustreError: Skipped 1 previous similar message [12258.332968] Lustre: Skipped 1 previous similar message [12263.392832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12263.402920] Lustre: Skipped 1 previous similar message [12264.433704] Lustre: server umount lustre-OST0000 complete [12268.515490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12268.534022] Lustre: Skipped 2 previous similar messages [12273.635421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12273.653391] Lustre: Skipped 3 previous similar messages [12274.515021] Lustre: server umount lustre-MDT0000 complete [12278.417799] LustreError: 336688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624800 with bad export cookie 9744952566856860451 [12278.421824] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12278.444243] LustreError: 336688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12278.841893] Lustre: server umount lustre-MDT0001 complete [12291.540167] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [12294.179988] Key type lgssc unregistered [12294.454054] LNet: 340144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12294.460274] LNetError: 340144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12294.477976] LNet: Removed LNI 192.168.203.143@tcp [12295.210923] Key type .llcrypt unregistered [12295.217191] Key type ._llcrypt unregistered [12314.026974] Key type ._llcrypt registered [12314.038879] Key type .llcrypt registered [12314.202273] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [12330.843561] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [12332.258424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12332.287647] alg: No test for adler32 (adler32-zlib) [12333.462619] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [12333.844558] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [12335.680191] Key type lgssc registered [12337.334457] Lustre: Echo OBD driver; http://www.lustre.org/ [12379.372271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12379.411618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12380.712987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12380.793643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12380.915674] Lustre: lustre-MDT0000: new disk, initializing [12380.998278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12381.024070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12386.027290] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [12399.157904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12399.373240] Lustre: 343304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12399.511437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12399.517054] Lustre: Skipped 1 previous similar message [12399.610364] Lustre: lustre-MDT0001: new disk, initializing [12399.726889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12399.798932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12399.811529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12405.673653] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [12418.619455] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12425.115663] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12432.974597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12433.228396] Lustre: lustre-OST0000: new disk, initializing [12433.232409] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12433.239429] Lustre: 345271:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12433.325942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12440.020646] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [12440.687027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12440.710107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12440.811707] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12450.216599] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12455.323836] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12455.553109] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12459.941829] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12460.121244] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12466.663867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12466.664593] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12466.670132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12466.675749] LustreError: Skipped 1 previous similar message [12466.699820] Lustre: Skipped 1 previous similar message [12471.779482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12471.802300] Lustre: Skipped 2 previous similar messages [12476.383392] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12476.558463] Lustre: server umount lustre-OST0000 complete [12481.505049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12481.524991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12481.557563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12486.508221] Lustre: server umount lustre-MDT0000 complete [12490.027109] LustreError: 343296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625011 with bad export cookie 4796894603448555097 [12490.029983] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12490.046922] LustreError: 343296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12490.450778] Lustre: server umount lustre-MDT0001 complete [12498.815703] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:16:58 (1788625018) [12516.018653] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [12525.858389] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [12561.772842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12562.010946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12562.032430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12562.134574] Lustre: lustre-MDT0000: new disk, initializing [12562.218566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12562.231892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12566.813026] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [12577.750058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12577.875548] Lustre: 350034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12577.943339] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12577.949827] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12577.951563] Lustre: Skipped 2 previous similar messages [12577.963812] Lustre: Skipped 1 previous similar message [12578.076702] Lustre: lustre-MDT0001: new disk, initializing [12578.240769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12578.293072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12578.308571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12583.058343] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [12592.436564] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12598.038579] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12604.467518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12604.750801] Lustre: lustre-OST0000: new disk, initializing [12604.757864] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12604.767256] Lustre: 352002:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12604.872050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12611.392912] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [12611.659311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12611.671199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12611.777837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12620.713251] Lustre: DEBUG MARKER: oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12658.148276] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12658.164190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12658.182100] Lustre: Skipped 2 previous similar messages [12658.182671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12658.207423] Lustre: Skipped 1 previous similar message [12660.022173] Lustre: server umount lustre-OST0000 complete [12665.314542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12665.333580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12668.384033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12668.395952] Lustre: Skipped 2 previous similar messages [12670.669374] Lustre: server umount lustre-MDT0000 complete [12673.514132] LustreError: 352593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12673.549769] LustreError: 352593:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12675.291049] LustreError: 350027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625197 with bad export cookie 4796894603448555979 [12675.296216] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12675.313580] LustreError: 350027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12675.832286] Lustre: server umount lustre-MDT0001 complete [12690.880472] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [12695.447754] Key type lgssc unregistered [12695.803814] LNet: 354155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12695.820209] LNetError: 354155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12695.855910] LNet: Removed LNI 192.168.203.143@tcp [12697.040310] Key type .llcrypt unregistered [12697.042571] Key type ._llcrypt unregistered [12798.066706] Key type ._llcrypt registered [12798.068782] Key type .llcrypt registered [12798.186823] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [12798.588554] Key type .llcrypt unregistered [12798.590579] Key type ._llcrypt unregistered [12873.536620] Key type ._llcrypt registered [12873.538787] Key type .llcrypt registered [12873.629565] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [12873.879464] Key type .llcrypt unregistered [12873.883189] Key type ._llcrypt unregistered [12904.878606] Key type ._llcrypt registered [12904.883228] Key type .llcrypt registered [12905.013190] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [12905.364474] Key type .llcrypt unregistered [12905.366343] Key type ._llcrypt unregistered [12936.798440] Key type ._llcrypt registered [12936.803404] Key type .llcrypt registered [12936.951353] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [12937.286844] Key type .llcrypt unregistered [12937.288707] Key type ._llcrypt unregistered [13012.283375] Key type ._llcrypt registered [13012.286222] Key type .llcrypt registered [13012.394619] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [13012.773840] Key type .llcrypt unregistered [13012.777804] Key type ._llcrypt unregistered [13043.750530] Key type ._llcrypt registered [13043.753234] Key type .llcrypt registered [13043.851740] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [13044.077434] Key type .llcrypt unregistered [13044.081827] Key type ._llcrypt unregistered [13077.765374] Key type ._llcrypt registered [13077.771324] Key type .llcrypt registered [13077.929594] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [13078.317977] Key type .llcrypt unregistered [13078.322437] Key type ._llcrypt unregistered [13110.494358] Key type ._llcrypt registered [13110.498260] Key type .llcrypt registered [13110.616757] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [13110.894413] Key type .llcrypt unregistered [13110.899343] Key type ._llcrypt unregistered [13143.726501] Key type ._llcrypt registered [13143.731837] Key type .llcrypt registered [13143.822795] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [13144.073626] Key type .llcrypt unregistered [13144.075932] Key type ._llcrypt unregistered [13176.197467] Key type ._llcrypt registered [13176.201712] Key type .llcrypt registered [13176.349595] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [13176.711273] Key type .llcrypt unregistered [13176.716841] Key type ._llcrypt unregistered [13242.859646] Key type ._llcrypt registered [13242.871754] Key type .llcrypt registered [13242.990686] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [13243.397979] Key type .llcrypt unregistered [13243.404893] Key type ._llcrypt unregistered [13272.425905] Key type ._llcrypt registered [13272.432693] Key type .llcrypt registered [13272.527211] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [13272.764890] Key type .llcrypt unregistered [13272.768175] Key type ._llcrypt unregistered [13303.625937] Key type ._llcrypt registered [13303.628150] Key type .llcrypt registered [13303.758749] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [13304.060954] Key type .llcrypt unregistered [13304.064906] Key type ._llcrypt unregistered [13358.876592] Key type ._llcrypt registered [13358.879902] Key type .llcrypt registered [13359.040345] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [13375.897823] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [13377.072083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13377.092066] alg: No test for adler32 (adler32-zlib) [13378.285099] Lustre: Lustre: Build Version: 2.17.58_2_ge9089f3 [13378.603405] LNet: Added LNI 192.168.203.143@tcp [8/256/0/180] [13380.297568] Key type lgssc registered [13381.720447] Lustre: Echo OBD driver; http://www.lustre.org/ [13420.210340] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13420.278980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13421.723509] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13421.797368] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13421.909212] Lustre: tst250fs-MDT0000: new disk, initializing [13422.072374] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13422.114715] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13427.244434] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [13439.272862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13439.438438] Lustre: 374471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13439.516805] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13439.521634] Lustre: Skipped 1 previous similar message [13439.621494] Lustre: tst250fs-MDT0001: new disk, initializing [13439.743569] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13439.807344] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13439.833895] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13444.557160] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [13449.352259] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13455.541759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13455.849435] Lustre: tst250fs-OST0000: new disk, initializing [13455.861273] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13455.876042] Lustre: 376104:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13455.972609] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13461.057782] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13461.067329] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13461.143705] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13461.712142] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [13473.563715] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13473.742790] Lustre: tst250fs-OST0001: new disk, initializing [13473.751097] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13473.755581] Lustre: 376975:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13473.888450] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13480.188834] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_default_debug -1 all [13482.088340] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13482.100552] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13482.245049] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13486.221542] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13488.526691] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13488.538327] Lustre: Skipped 1 previous similar message [13511.649022] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13511.659926] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13511.683467] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13513.184969] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13513.193823] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13513.224350] Lustre: Skipped 2 previous similar messages [13516.878409] Lustre: server umount tst250fs-MDT0000 complete [13521.284226] LustreError: 374462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626043 with bad export cookie 14145309154939072873 [13521.317168] LustreError: 374462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13521.334145] LustreError: MGC192.168.203.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13521.991079] Lustre: server umount tst250fs-MDT0001 complete [13536.269738] Lustre: server umount tst250fs-OST0000 complete [13539.168048] Lustre: 372694:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788626045/real 1788626045] req@ffff8941c72c4a80 x1875510191785856/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788626061 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13539.210644] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13540.427966] Lustre: server umount tst250fs-OST0001 complete [13566.112535] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [13575.049444] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [13613.480880] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:35:33 (1788626133) [13616.131621] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13634.418861] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing set_hostid [13646.064594] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing load_modules_local [13680.570234] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13401 sec ========= 12:36:40 (1788626200) [13682.696803] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:36:42 (1788626202) === [13686.679933] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:36:46 (1788626206) === [13728.969276] Lustre: DEBUG MARKER: oleg343-server.virtnet: executing unload_modules_local [13731.975256] Key type lgssc unregistered [13732.228524] LNet: 386118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13732.239777] LNetError: 386118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13732.265746] LNet: Removed LNI 192.168.203.143@tcp [13733.505205] Key type .llcrypt unregistered [13733.509956] Key type ._llcrypt unregistered