[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 494856293 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001019] APIC: Switch to symmetric I/O mode setup [ 0.003393] x2apic enabled [ 0.004013] Switched APIC routing to physical x2apic. [ 0.005019] kvm-guest: setup PV IPIs [ 0.008956] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010021] pid_max: default: 32768 minimum: 301 [ 0.011171] LSM: Security Framework initializing [ 0.012078] Yama: becoming mindful. [ 0.013060] SELinux: Initializing. [ 0.014097] *** VALIDATE selinux *** [ 0.022318] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027028] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028210] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029138] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030156] *** VALIDATE tmpfs *** [ 0.031580] *** VALIDATE proc *** [ 0.032353] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035341] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036202] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037017] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039038] Spectre V2 : User space: Vulnerable [ 0.040015] Speculative Store Bypass: Vulnerable [ 0.043568] debug: unmapping init [mem 0xffffffff9c259000-0xffffffff9c260fff] [ 0.045229] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046810] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047035] ... version: 2 [ 0.048022] ... bit width: 48 [ 0.049023] ... generic registers: 4 [ 0.050023] ... value mask: 0000ffffffffffff [ 0.051023] ... max period: 00007fffffffffff [ 0.052026] ... fixed-purpose events: 3 [ 0.053020] ... event mask: 000000070000000f [ 0.054382] rcu: Hierarchical SRCU implementation. [ 0.056700] smp: Bringing up secondary CPUs ... [ 0.057676] x86: Booting SMP configuration: [ 0.058038] .... node #0, CPUs: #1 #2 #3 [ 0.061427] smp: Brought up 1 node, 4 CPUs [ 0.063019] smpboot: Max logical packages: 1 [ 0.064026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.229031] node 0 deferred pages initialised in 163ms [ 0.232166] devtmpfs: initialized [ 0.233261] x86/mm: Memory block size: 128MB [ 0.235833] gcov: version magic: 0x41383552 [ 0.237554] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.243091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.247390] pinctrl core: initialized pinctrl subsystem [ 0.249215] [ 0.250008] ************************************************************* [ 0.254017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.257012] ** ** [ 0.261015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.264014] ** ** [ 0.268016] ** This means that this kernel is built to expose internal ** [ 0.271015] ** IOMMU data structures, which may compromise security on ** [ 0.274014] ** your system. ** [ 0.277017] ** ** [ 0.280014] ** If you see this message and you are not debugging the ** [ 0.283014] ** kernel, report this immediately to your vendor! ** [ 0.286030] ** ** [ 0.289016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.293018] ************************************************************* [ 0.296826] NET: Registered protocol family 16 [ 0.299478] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.303122] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.305084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.309104] cpuidle: using governor menu [ 0.310982] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.313623] PCI: Using configuration type 1 for base access [ 0.316142] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.326267] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.328039] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.333055] cryptd: max_cpu_qlen set to 1000 [ 0.335263] ACPI: Added _OSI(Module Device) [ 0.337012] ACPI: Added _OSI(Processor Device) [ 0.338013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.340012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.344977] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.351489] ACPI: Interpreter enabled [ 0.353116] ACPI: PM: (supports S0 S3 S4 S5) [ 0.355024] ACPI: Using IOAPIC for interrupt routing [ 0.357204] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.361333] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.371169] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.374045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.377022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.382087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.387479] acpiphp: Slot [2] registered [ 0.388164] acpiphp: Slot [5] registered [ 0.390176] acpiphp: Slot [6] registered [ 0.391155] acpiphp: Slot [7] registered [ 0.393190] acpiphp: Slot [8] registered [ 0.394215] acpiphp: Slot [9] registered [ 0.396198] acpiphp: Slot [10] registered [ 0.398186] acpiphp: Slot [3] registered [ 0.399116] acpiphp: Slot [4] registered [ 0.400163] acpiphp: Slot [11] registered [ 0.402132] acpiphp: Slot [12] registered [ 0.404157] acpiphp: Slot [13] registered [ 0.406115] acpiphp: Slot [14] registered [ 0.407126] acpiphp: Slot [15] registered [ 0.408083] acpiphp: Slot [16] registered [ 0.410112] acpiphp: Slot [17] registered [ 0.411054] acpiphp: Slot [18] registered [ 0.412058] acpiphp: Slot [19] registered [ 0.412957] acpiphp: Slot [20] registered [ 0.414071] acpiphp: Slot [21] registered [ 0.415049] acpiphp: Slot [22] registered [ 0.416038] acpiphp: Slot [23] registered [ 0.417070] acpiphp: Slot [24] registered [ 0.418053] acpiphp: Slot [25] registered [ 0.418964] acpiphp: Slot [26] registered [ 0.420085] acpiphp: Slot [27] registered [ 0.421106] acpiphp: Slot [28] registered [ 0.423087] acpiphp: Slot [29] registered [ 0.424055] acpiphp: Slot [30] registered [ 0.425109] acpiphp: Slot [31] registered [ 0.426056] PCI host bridge to bus 0000:00 [ 0.427016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.429021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.431017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.433019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.435044] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.438048] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.440211] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.444191] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.448562] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.458883] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.463000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.466052] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.469030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.473028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.476020] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.478488] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.480111] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.485011] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.489012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.500013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.503012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.508611] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.514020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.521023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.541020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.553540] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.572022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.589024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.605026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.612329] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.623019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.640030] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.664018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.676513] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.685024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.693022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.717024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.725477] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.733013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.745019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.763020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.780436] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.789023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.798022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.817025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.831000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.833575] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.836526] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.839546] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.842289] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.847294] iommu: Default domain type: Passthrough [ 0.849419] SCSI subsystem initialized [ 0.850177] ACPI: bus type USB registered [ 0.851099] usbcore: registered new interface driver usbfs [ 0.853091] usbcore: registered new interface driver hub [ 0.854074] usbcore: registered new device driver usb [ 0.869131] pps_core: LinuxPPS API ver. 1 registered [ 0.870009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.872059] PTP clock support registered [ 0.874126] EDAC MC: Ver: 3.0.0 [ 0.875349] PCI: Using ACPI for IRQ routing [ 0.876692] NetLabel: Initializing [ 0.877011] NetLabel: domain hash size = 128 [ 0.878005] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.879128] NetLabel: unlabeled traffic allowed by default [ 0.882107] vgaarb: loaded [ 0.883316] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.884012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.889668] clocksource: Switched to clocksource kvm-clock [ 0.984921] VFS: Disk quotas dquot_6.6.0 [ 0.986406] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.988210] *** VALIDATE ramfs *** [ 0.989046] *** VALIDATE hugetlbfs *** [ 0.990471] pnp: PnP ACPI init [ 0.992600] pnp: PnP ACPI: found 6 devices [ 1.006641] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.011215] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.013806] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.016805] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.020525] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.023278] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.026096] NET: Registered protocol family 2 [ 1.027955] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.033825] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.036693] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.042968] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.047462] TCP: Hash tables configured (established 65536 bind 65536) [ 1.050762] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.054592] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.057940] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.061326] NET: Registered protocol family 1 [ 1.064503] RPC: Registered named UNIX socket transport module. [ 1.067212] RPC: Registered udp transport module. [ 1.068838] RPC: Registered tcp transport module. [ 1.070152] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.071919] NET: Registered protocol family 44 [ 1.073101] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.075562] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.077168] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.079181] PCI: CLS 0 bytes, default 64 [ 1.081283] Unpacking initramfs... [ 2.496963] debug: unmapping init [mem 0xffffa0813cc54000-0xffffa0813ffbffff] [ 2.501276] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.503589] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.506564] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.003627] Initialise system trusted keyrings [ 3.005570] Key type blacklist registered [ 3.007685] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.017234] zbud: loaded [ 3.022639] *** VALIDATE nfs *** [ 3.023739] *** VALIDATE nfs4 *** [ 3.025364] pstore: using deflate compression [ 3.028756] Platform Keyring initialized [ 3.137422] NET: Registered protocol family 38 [ 3.139411] Key type asymmetric registered [ 3.141159] Asymmetric key parser 'x509' registered [ 3.143134] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.147776] io scheduler mq-deadline registered [ 3.149741] io scheduler kyber registered [ 3.151742] io scheduler bfq registered [ 3.154291] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.158276] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.161949] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.165135] ACPI: Power Button [PWRF] [ 3.171629] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.179194] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.191190] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.200456] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.219330] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.246853] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.272051] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.276535] Non-volatile memory driver v1.3 [ 3.277936] Linux agpgart interface v0.103 [ 3.309177] virtio_blk virtio1: [vda] 149792 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.312392] vda: detected capacity change from 0 to 76693504 [ 3.328749] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.331791] vdb: detected capacity change from 0 to 1073741824 [ 3.349913] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.353428] vdc: detected capacity change from 0 to 2621440000 [ 3.370683] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.373570] vdd: detected capacity change from 0 to 2621440000 [ 3.390421] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.393596] vde: detected capacity change from 0 to 4294967296 [ 3.410900] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.414359] vdf: detected capacity change from 0 to 4294967296 [ 3.427712] libphy: Fixed MDIO Bus: probed [ 3.444609] usbcore: registered new interface driver usbserial_generic [ 3.447113] usbserial: USB Serial support registered for generic [ 3.449379] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.453578] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.455361] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.458135] mousedev: PS/2 mouse device common for all mice [ 3.461635] rtc_cmos 00:05: RTC can wake from S4 [ 3.465044] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.466085] rtc_cmos 00:05: registered as rtc0 [ 3.472442] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.472645] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.477791] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.479745] intel_pstate: CPU model not supported [ 3.489489] hid: raw HID events driver (C) Jiri Kosina [ 3.491472] usbcore: registered new interface driver usbhid [ 3.492896] usbhid: USB HID core driver [ 3.494412] drop_monitor: Initializing network drop monitor service [ 3.497089] Initializing XFRM netlink socket [ 3.499263] NET: Registered protocol family 10 [ 3.502279] Segment Routing with IPv6 [ 3.503922] NET: Registered protocol family 17 [ 3.506278] mpls_gso: MPLS GSO support [ 3.511385] RAS: Correctable Errors collector initialized. [ 3.513503] AVX version of gcm_enc/dec engaged. [ 3.514970] AES CTR mode by8 optimization enabled [ 3.616153] sched_clock: Marking stable (3616121663, 0)->(4610123568, -994001905) [ 3.620673] registered taskstats version 1 [ 3.623335] Loading compiled-in X.509 certificates [ 3.625425] zswap: loaded using pool lzo/zbud [ 3.650645] Key type big_key registered [ 3.662101] Key type encrypted registered [ 3.663603] ima: No TPM chip found, activating TPM-bypass! [ 3.665329] ima: Allocated hash algorithm: sha1 [ 3.666625] ima: No architecture policies found [ 3.668249] evm: Initialising EVM extended attributes: [ 3.669860] evm: security.selinux [ 3.670931] evm: security.ima [ 3.671891] evm: security.capability [ 3.673164] evm: HMAC attrs: 0x1 [ 3.676114] rtc_cmos 00:05: setting system clock to 2026-09-03 20:47:39 UTC (1788468459) [ 3.683303] debug: unmapping init [mem 0xffffffff9d203000-0xffffffff9d3fffff] [ 3.686025] debug: unmapping init [mem 0xffffffff9bf82000-0xffffffff9c258fff] [ 3.696174] Write protecting the kernel read-only data: 28672k [ 3.698217] debug: unmapping init [mem 0xffffffff9a603000-0xffffffff9a7fffff] [ 3.699813] debug: unmapping init [mem 0xffffffff9af14000-0xffffffff9affffff] [ 3.735305] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.741166] systemd[1]: Detected virtualization kvm. [ 3.743677] systemd[1]: Detected architecture x86-64. [ 3.746261] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.775186] systemd[1]: No hostname configured. [ 3.777427] systemd[1]: Set hostname to . [ 3.779819] random: systemd: uninitialized urandom read (16 bytes read) [ 3.783114] systemd[1]: Initializing machine ID from random generator. [ 3.828485] random: ln: uninitialized urandom read (6 bytes read) [ 3.928937] random: systemd: uninitialized urandom read (16 bytes read) [ 3.931669] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.937705] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.947491] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.586185] device-mapper: uevent: version 1.0.3 [ 4.587846] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.341580] virtio_net virtio0 ens2: renamed from eth0 [ 5.357572] random: fast init done [ 5.406238] scsi host0: ata_piix [ 5.439319] scsi host1: ata_piix [ 5.440934] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.443652] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.805669] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.178566] random: crng init done [ 10.180068] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.634042] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.040892] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.275059] SELinux: Disabled at runtime. [ 13.504736] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.537268] systemd[1]: Detected virtualization kvm. [ 13.540828] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.188068] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.196938] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.207712] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.218569] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.227506] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.234440] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.244956] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ 15.358414] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 16.652613] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.902751] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.948113] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.323519] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.416409] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 23.689253] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 24.881610] NFS: Registering the id_resolver key type [ 24.892825] Key type id_resolver registered [ 24.894273] Key type id_legacy registered [ ***] A start job is running for Configur…-only root support (9s / no limit) [ **] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg123-server login: [ 93.744498] libcfs: loading out-of-tree module taints kernel. [ 93.845631] Key type ._llcrypt registered [ 93.855901] Key type .llcrypt registered [ 93.953712] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 118.888729] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 121.396677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 121.422686] alg: No test for adler32 (adler32-zlib) [ 122.854400] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 123.638957] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 125.623167] Key type lgssc registered [ 127.552952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 146.769232] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 185.870817] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 197.760079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 197.796727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 199.048411] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 199.083483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 199.226857] Lustre: lustre-MDT0000: new disk, initializing [ 199.346242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.370338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 203.918835] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 220.369732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 220.511587] Lustre: 6505:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 220.590544] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 220.595262] Lustre: Skipped 1 previous similar message [ 220.721358] Lustre: lustre-MDT0001: new disk, initializing [ 220.824887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 220.885416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 220.901163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 226.282300] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.510431] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 241.632748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 241.931538] Lustre: lustre-OST0000: new disk, initializing [ 241.937185] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 241.945580] Lustre: 8443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 242.031345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.589619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 242.601613] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 242.719603] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 249.217777] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 263.617829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 263.794610] Lustre: lustre-OST0001: new disk, initializing [ 263.796980] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 263.802713] Lustre: 9515:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 263.866368] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 269.913419] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 273.540268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 273.554479] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 273.640401] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 274.314211] hrtimer: interrupt took 3203011 ns [ 282.601087] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 296.462593] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 305.328726] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing check_logdir /tmp/testlogs/ [ 311.146938] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing yml_node [ 315.481250] Lustre: DEBUG MARKER: Client: 2.17.58.4 [ 318.338812] Lustre: DEBUG MARKER: MDS: 2.17.58.4 [ 320.762913] Lustre: DEBUG MARKER: OSS: 2.17.58.4 [ 322.511242] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 16:52:56 EDT 2026 [ 340.974856] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 342.684381] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 359.393030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 359.408205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.443246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 360.931137] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 360.935473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 360.955370] Lustre: Skipped 1 previous similar message [ 360.975073] Lustre: Skipped 3 previous similar messages [ 365.061699] Lustre: server umount lustre-MDT0000 complete [ 368.673834] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788468824 with bad export cookie 4304486797952844253 [ 368.677695] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 368.682423] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 368.709074] Lustre: 12343:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 371.170041] LustreError: 6512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 371.175172] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 371.175571] Lustre: 12345:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 371.176033] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 371.185388] LustreError: 6512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 371.201452] Lustre: Skipped 2 previous similar messages [ 375.272019] Lustre: server umount lustre-MDT0001 complete [ 376.287943] Lustre: 12343:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 379.330380] Lustre: server umount lustre-OST0000 complete [ 381.411222] Lustre: 12344:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 383.201846] Lustre: server umount lustre-OST0001 complete [ 390.738844] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 401.038330] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 446.626648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.034526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 447.114139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 447.373870] Lustre: lustre-MDT0000: new disk, initializing [ 447.507817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 447.537238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 452.313623] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 467.284144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 467.358768] Lustre: 15076:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 467.440426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 467.448191] Lustre: Skipped 1 previous similar message [ 467.552654] Lustre: lustre-MDT0001: new disk, initializing [ 467.832305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 468.007054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 468.029587] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 475.277818] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 488.111889] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 493.726374] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 501.495502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 501.890901] Lustre: lustre-OST0000: new disk, initializing [ 501.901773] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 501.915764] Lustre: 17046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 502.050932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 503.356124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 503.369970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 503.462381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 508.555446] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 518.251457] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 523.237837] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 523.558877] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 528.271417] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 528.509611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 534.498175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 534.498844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.536421] LustreError: Skipped 1 previous similar message [ 534.540396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 534.540405] Lustre: Skipped 1 previous similar message [ 534.614939] Lustre: Skipped 1 previous similar message [ 539.673202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 539.679466] Lustre: Skipped 2 previous similar messages [ 545.248953] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 545.556499] Lustre: server umount lustre-OST0000 complete [ 554.978316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 555.003621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 555.011380] Lustre: Skipped 3 previous similar messages [ 560.025592] Lustre: server umount lustre-MDT0000 complete [ 560.100109] LustreError: 15921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.121186] LustreError: 15921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 564.665887] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469020 with bad export cookie 4304486797952845884 [ 564.679618] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 564.686115] LustreError: 15067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 564.720742] Lustre: 18638:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 565.321490] Lustre: server umount lustre-MDT0001 complete [ 575.332988] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 16:57:08 (1788469028) [ 602.811868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 603.706406] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 603.948226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 609.250753] LustreError: 19930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.644658] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 613.356087] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.709090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 627.261628] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 636.348390] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 643.539385] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 646.049868] Lustre: Failing over lustre-MDT0000 [ 646.065109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 646.083189] Lustre: Skipped 1 previous similar message [ 646.094437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 646.098724] Lustre: Skipped 1 previous similar message [ 646.223324] Lustre: server umount lustre-MDT0000 complete [ 647.137686] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 647.151494] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 652.257533] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 652.277321] Lustre: 21573:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 666.595697] LustreError: 19930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 666.633173] LustreError: 19930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 668.992343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.361900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 669.371821] Lustre: Skipped 1 previous similar message [ 669.429853] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 674.790099] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 674.825440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 674.896568] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 675.368852] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 686.606964] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 695.264731] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 695.268182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 695.305279] Lustre: Skipped 1 previous similar message [ 695.315213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 697.754691] Lustre: server umount lustre-MDT0000 complete [ 701.428289] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 701.434452] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 701.450026] LustreError: 19929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 701.499157] Lustre: 22654:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 701.532832] Lustre: 22654:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 3 previous similar messages [ 703.703793] Lustre: server umount lustre-MDT0001 complete [ 714.901975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.325912] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 735.446995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.641701] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 751.488836] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 759.232534] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 761.366320] Lustre: Failing over lustre-MDT0000 [ 761.832631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 761.839315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 761.849336] Lustre: Skipped 1 previous similar message [ 761.975555] Lustre: server umount lustre-MDT0000 complete [ 765.923079] LustreError: 23274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 765.925986] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 765.951253] LustreError: 23274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 765.978329] Lustre: 24925:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 786.814143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 787.687038] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 793.060127] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 793.087939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 793.099399] Lustre: Skipped 1 previous similar message [ 793.131194] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 794.959474] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 806.873274] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 813.558042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 813.579771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 813.613168] Lustre: Skipped 2 previous similar messages [ 817.731532] Lustre: server umount lustre-MDT0000 complete [ 820.706022] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 823.367653] Lustre: server umount lustre-MDT0001 complete [ 837.820128] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 17:01:30 (1788469290) [ 859.089398] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 871.743268] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 917.137654] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 929.702855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 930.059351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 930.092055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 930.197755] Lustre: lustre-MDT0000: new disk, initializing [ 930.267646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 930.274455] Lustre: Skipped 3 previous similar messages [ 930.301133] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 934.686938] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 947.311718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 947.449751] Lustre: 29677:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 947.710528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 952.956663] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 964.028897] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 971.890512] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 981.079531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 981.395277] Lustre: 31648:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 983.174611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 983.216765] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 983.384429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 983.429053] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 988.330186] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 999.509940] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1005.030131] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1029.601761] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1029.603528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.621940] LustreError: Skipped 1 previous similar message [ 1029.622974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1029.622978] Lustre: Skipped 2 previous similar messages [ 1029.680799] Lustre: Skipped 4 previous similar messages [ 1030.992686] Lustre: server umount lustre-OST0000 complete [ 1044.360034] Lustre: server umount lustre-MDT0000 complete [ 1044.963040] LustreError: 31369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1044.999248] LustreError: 31369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 1049.294478] LustreError: 29671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469505 with bad export cookie 4304486797952848775 [ 1049.304675] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1049.311545] LustreError: 29671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1049.329096] Lustre: 32912:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1049.360258] Lustre: 32912:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 5 previous similar messages [ 1050.100614] Lustre: server umount lustre-MDT0001 complete [ 1067.467678] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 1071.015321] Key type lgssc unregistered [ 1071.524879] LNet: 33656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1071.531329] LNetError: 33656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1071.546312] LNet: Removed LNI 192.168.201.123@tcp [ 1073.748842] Key type .llcrypt unregistered [ 1073.754939] Key type ._llcrypt unregistered [ 1103.172666] Key type ._llcrypt registered [ 1103.178677] Key type .llcrypt registered [ 1103.292898] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1104.541879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1104.586387] alg: No test for adler32 (adler32-zlib) [ 1105.692212] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 1105.916154] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 1107.625099] Key type lgssc registered [ 1108.612636] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1116.536899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1116.557827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1118.095274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1124.295426] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1127.397993] LustreError: 35064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.423138] LustreError: 35064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1132.517966] LustreError: 35065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.026197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1135.620540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1140.736707] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1150.748113] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1161.477816] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1169.232307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1169.733663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1178.996504] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1181.061407] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1188.871497] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1193.478842] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1213.505600] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 17:07:46 (1788469666) [ 1216.726394] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1218.879194] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1226.855883] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 17:08:00 (1788469680) [ 1237.294799] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 17:08:11 (1788469691) [ 1238.614561] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1249.464791] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1249.473464] Lustre: Skipped 2 previous similar messages [ 1251.660757] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 17:08:25 (1788469705) [ 1255.048667] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1255.059867] Lustre: Skipped 2 previous similar messages [ 1263.587225] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1263.601996] Lustre: Skipped 1 previous similar message [ 1281.231199] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 17:08:55 (1788469735) [ 1284.551072] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1288.607415] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1298.623334] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 17:09:12 (1788469752) [ 1300.544595] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1300.549888] Lustre: Skipped 9 previous similar messages [ 1314.352724] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 17:09:28 (1788469768) [ 1331.686866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1331.691624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1331.703205] Lustre: Skipped 1 previous similar message [ 1336.690433] Lustre: server umount lustre-OST0000 complete [ 1341.920985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.931446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1341.939448] Lustre: Skipped 1 previous similar message [ 1347.046775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1347.064338] Lustre: Skipped 1 previous similar message [ 1348.579416] Lustre: server umount lustre-MDT0000 complete [ 1352.172529] LustreError: 36048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.184906] LustreError: 36048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1352.591321] LustreError: 35046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788469808 with bad export cookie 3179470823494082874 [ 1352.592052] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1352.599385] LustreError: 35046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1353.186131] Lustre: server umount lustre-MDT0001 complete [ 1366.015136] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 1368.854511] Key type lgssc unregistered [ 1369.129237] LNet: 42927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1369.137631] LNetError: 42927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1370.215497] LNet: Removed LNI 192.168.201.123@tcp [ 1371.136066] Key type .llcrypt unregistered [ 1371.138789] Key type ._llcrypt unregistered [ 1392.104655] Key type ._llcrypt registered [ 1392.106569] Key type .llcrypt registered [ 1392.222850] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 1409.131186] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 1410.517725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1410.582395] alg: No test for adler32 (adler32-zlib) [ 1411.639814] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 1411.863579] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 1413.559177] Key type lgssc registered [ 1415.101200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1463.477869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1463.579229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1464.910209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1464.990473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1465.111503] Lustre: lustre-MDT0000: new disk, initializing [ 1465.266544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1465.294455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1470.008961] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1481.625309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1481.753809] Lustre: 46082:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1481.901685] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1481.909122] Lustre: Skipped 1 previous similar message [ 1482.027669] Lustre: lustre-MDT0001: new disk, initializing [ 1482.223665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1482.282140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1482.324140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1487.810715] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1499.268395] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1508.969712] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1519.337655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1519.806970] Lustre: lustre-OST0000: new disk, initializing [ 1519.821551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1519.837961] Lustre: 48051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1520.058716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1528.933448] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1529.559144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1529.582399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1529.762640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1540.953476] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1547.388138] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1547.879759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1555.791629] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1556.220821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1560.544025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1560.550097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.568835] LustreError: Skipped 1 previous similar message [ 1560.607851] Lustre: Skipped 1 previous similar message [ 1560.619970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1560.653685] Lustre: Skipped 1 previous similar message [ 1565.812830] Lustre: server umount lustre-OST0000 complete [ 1574.369796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1574.385456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1574.410030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1575.955199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1575.956893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1579.131275] Lustre: server umount lustre-MDT0000 complete [ 1581.033157] LustreError: 46095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1581.078487] LustreError: 46095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1585.381889] LustreError: 46076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470041 with bad export cookie 11198720781829292193 [ 1585.415263] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1586.153896] Lustre: server umount lustre-MDT0001 complete [ 1599.143129] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 17:14:13 (1788470053) [ 1607.965941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1608.485455] LustreError: 50515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1608.565355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1613.792640] LustreError: 50516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.242992] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1618.916662] LustreError: 50515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1624.035286] LustreError: 50516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.014995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1625.557092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1632.256765] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1642.714088] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1650.306528] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1658.418419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1658.833977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1666.469782] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1677.565638] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1684.477465] LustreError: 53046:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1686.386536] Lustre: Modifying parameter general.timeout=1 in log params [ 1687.852826] Lustre: Modifying parameter general.timeout=2 in log params [ 1690.920883] Lustre: Modifying parameter general.timeout=4 in log params [ 1690.937500] Lustre: Skipped 1 previous similar message [ 1696.347184] Lustre: Modifying parameter general.timeout=7 in log params [ 1696.364626] Lustre: Skipped 2 previous similar messages [ 1704.799917] Lustre: Modifying parameter general.timeout=12 in log params [ 1704.811729] Lustre: Skipped 4 previous similar messages [ 1721.394928] Lustre: Modifying parameter general.timeout=28 in log params [ 1721.402180] Lustre: Skipped 15 previous similar messages [ 1753.700904] Lustre: Modifying parameter general.timeout=57 in log params [ 1753.704646] Lustre: Skipped 28 previous similar messages [ 1817.843205] Lustre: Modifying parameter general.timeout=106 in log params [ 1817.872733] Lustre: Skipped 48 previous similar messages [ 1859.920222] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 17:18:33 (1788470313) [ 1863.090900] Lustre: 60020:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1864.987846] Lustre: 60120:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1867.281581] Lustre: 60218:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1871.405304] Lustre: 60429:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1871.414031] Lustre: 60429:0:(mgs_llog.c:1456:mgs_modify_param()) Skipped 1 previous similar message [ 1906.486039] Lustre: 61894:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1906.498254] Lustre: 61894:0:(mgs_llog.c:1456:mgs_modify_param()) Skipped 1 previous similar message [ 1906.515275] LustreError: 61894:0:(mgs_handler.c:1215:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1911.448625] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 17:19:25 (1788470365) [ 1927.649266] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1927.652600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.678660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1927.685480] Lustre: Skipped 1 previous similar message [ 1932.542871] Lustre: server umount lustre-MDT0000 complete [ 1934.830663] LustreError: 51243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1934.850745] LustreError: 51243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1936.035756] LustreError: 50496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470391 with bad export cookie 11198720781829293082 [ 1936.048476] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1936.055097] LustreError: 50496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 1936.467076] Lustre: server umount lustre-MDT0001 complete [ 1945.376347] Lustre: 62916:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1950.772661] Lustre: server umount lustre-OST0000 complete [ 1981.265110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1981.569246] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1981.598377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1981.607558] Lustre: Skipped 46 previous similar messages [ 1981.886941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1986.408289] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 1996.037532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1996.123033] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1996.144224] Lustre: 64435:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2001.503983] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2009.908476] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2016.884962] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2028.924469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2029.286136] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2029.966749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2029.989256] Lustre: Skipped 1 previous similar message [ 2041.334262] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2053.612946] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2055.817207] Lustre: 67003:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2068.960720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2068.974963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2068.998656] Lustre: Skipped 2 previous similar messages [ 2069.011613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2069.020702] Lustre: Skipped 3 previous similar messages [ 2071.377145] Lustre: server umount lustre-OST0000 complete [ 2078.177239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2078.182765] LustreError: Skipped 1 previous similar message [ 2078.192587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2078.215757] Lustre: Skipped 1 previous similar message [ 2078.224588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2078.227297] Lustre: Skipped 1 previous similar message [ 2081.980492] Lustre: server umount lustre-MDT0000 complete [ 2084.324920] LustreError: 64444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2084.356492] LustreError: 64444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2085.817319] LustreError: 64425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470541 with bad export cookie 11198720781829296540 [ 2085.823116] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2085.827350] LustreError: 64425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2086.187640] Lustre: server umount lustre-MDT0001 complete [ 2098.632794] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2101.364214] Key type lgssc unregistered [ 2101.745410] LNet: 68214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.772452] LNetError: 68214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2101.794368] LNet: Removed LNI 192.168.201.123@tcp [ 2102.625743] Key type .llcrypt unregistered [ 2102.633543] Key type ._llcrypt unregistered [ 2131.940362] Key type ._llcrypt registered [ 2131.942874] Key type .llcrypt registered [ 2132.082380] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2133.317412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2133.571329] alg: No test for adler32 (adler32-zlib) [ 2134.793286] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 2135.100818] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2136.791526] Key type lgssc registered [ 2138.126894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2146.573813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2146.587630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.251514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2153.086359] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2157.552247] LustreError: 69625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2157.586287] LustreError: 69625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2162.668170] LustreError: 69624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.921366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2163.713980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2168.977224] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2178.542277] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2185.214987] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2192.807507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2193.337645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2200.510942] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2210.145560] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2230.037597] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2240.488038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2240.503644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.521553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2243.554917] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.555717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2243.567181] Lustre: Skipped 1 previous similar message [ 2243.604508] Lustre: Skipped 2 previous similar messages [ 2246.860594] Lustre: server umount lustre-MDT0000 complete [ 2248.677172] LustreError: 71416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.699590] LustreError: 71416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2251.111734] LustreError: 70365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470706 with bad export cookie 3702513747806372217 [ 2251.124866] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2251.378650] Lustre: server umount lustre-MDT0001 complete [ 2260.192933] Lustre: 73062:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2265.573537] Lustre: server umount lustre-OST0000 complete [ 2297.864649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.365474] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2298.418378] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2298.758087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.700268] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2314.464987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2314.596359] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2314.663372] Lustre: 74523:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2315.331635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2320.634624] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2329.678297] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2337.237481] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2345.121063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2345.378477] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2346.012726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2354.120725] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2364.775246] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2366.909231] Lustre: 77104:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2369.654909] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2380.776639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2380.781454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2380.800739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2382.774974] Lustre: server umount lustre-OST0000 complete [ 2391.033687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2391.046215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2391.048340] Lustre: Skipped 2 previous similar messages [ 2391.076203] Lustre: Skipped 2 previous similar messages [ 2393.659895] Lustre: server umount lustre-MDT0000 complete [ 2396.132985] LustreError: 75661:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2396.160914] LustreError: 75661:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2398.016933] LustreError: 74515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788470853 with bad export cookie 3702513747806373106 [ 2398.020409] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2398.029517] LustreError: 74515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2398.666783] Lustre: server umount lustre-MDT0001 complete [ 2414.353538] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2418.830753] Key type lgssc unregistered [ 2419.220398] LNet: 78313:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2419.236697] LNetError: 78313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2419.269640] LNet: Removed LNI 192.168.201.123@tcp [ 2420.546456] Key type .llcrypt unregistered [ 2420.554543] Key type ._llcrypt unregistered [ 2454.250692] Key type ._llcrypt registered [ 2454.252528] Key type .llcrypt registered [ 2454.380985] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2455.422713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2455.539469] alg: No test for adler32 (adler32-zlib) [ 2456.866706] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 2457.107232] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2458.775204] Key type lgssc registered [ 2460.727474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2470.476971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2470.532438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2472.186060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2477.645774] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2481.634776] LustreError: 79726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.659243] LustreError: 79726:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2486.752987] LustreError: 79727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.587221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2489.615082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2496.475903] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2507.572514] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2515.890472] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2524.884943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2525.620961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2533.027509] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2542.872541] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2549.788672] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2550.978870] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2551.983233] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2554.209629] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2554.222411] Lustre: Skipped 1 previous similar message [ 2558.473792] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2558.481146] Lustre: Skipped 3 previous similar messages [ 2566.776708] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2566.783378] Lustre: Skipped 6 previous similar messages [ 2583.805663] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2583.812207] Lustre: Skipped 13 previous similar messages [ 2616.783501] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2616.788736] Lustre: Skipped 24 previous similar messages [ 2676.663781] Lustre: 87839:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2680.003313] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 17:32:13 (1788471133) [ 2682.548585] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2685.018527] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 17:32:18 (1788471138) [ 2690.531289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.533931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2690.557625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2690.591785] Lustre: Skipped 1 previous similar message [ 2695.661451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2695.665689] Lustre: Skipped 1 previous similar message [ 2696.769529] Lustre: server umount lustre-OST0000 complete [ 2704.353251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2704.364182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2704.382435] Lustre: Skipped 1 previous similar message [ 2704.392354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2705.905947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2707.754726] Lustre: server umount lustre-MDT0000 complete [ 2711.011295] LustreError: 79726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.057791] LustreError: 79726:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2711.956426] LustreError: 79708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471167 with bad export cookie 18193702154246610631 [ 2711.959455] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2711.979908] LustreError: 79708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2712.610963] Lustre: server umount lustre-MDT0001 complete [ 2727.713470] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2731.161912] Key type lgssc unregistered [ 2731.531224] LNet: 89091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2731.549193] LNetError: 89091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2731.593264] LNet: Removed LNI 192.168.201.123@tcp [ 2733.185819] Key type .llcrypt unregistered [ 2733.194723] Key type ._llcrypt unregistered [ 2738.598879] Key type ._llcrypt registered [ 2738.624486] Key type .llcrypt registered [ 2738.967497] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2744.296480] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules [ 2745.330511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2745.355207] alg: No test for adler32 (adler32-zlib) [ 2746.516859] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 2746.739979] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2748.392548] Key type lgssc registered [ 2749.967131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2751.884757] LustreError: 90290:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2771.970688] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2774.389412] LustreError: 90290:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2782.623892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2782.648379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2784.509534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2789.776707] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2794.984835] LustreError: 91319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2795.011033] LustreError: 91319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2799.709303] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 17:34:13 (1788471253) [ 2800.098919] LustreError: 91320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2804.793396] LustreError: 91351:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2804.809590] LustreError: 91351:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2804.819295] LustreError: 91351:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2805.384973] Lustre: server umount lustre-MDT0000 complete [ 2820.281468] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 2824.058153] Key type lgssc unregistered [ 2824.481783] LNet: 92843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2824.501513] LNetError: 92843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2824.535347] LNet: Removed LNI 192.168.201.123@tcp [ 2826.109729] Key type .llcrypt unregistered [ 2826.112209] Key type ._llcrypt unregistered [ 2851.866842] Key type ._llcrypt registered [ 2851.877069] Key type .llcrypt registered [ 2852.043962] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 2853.161242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2853.202797] alg: No test for adler32 (adler32-zlib) [ 2854.421965] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 2854.668614] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 2856.303216] Key type lgssc registered [ 2857.495471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2865.785988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2865.814734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.301935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.506090] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2877.922302] LustreError: 94060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.954618] LustreError: 94060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2881.505743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2882.621985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2888.192430] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2896.690628] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2902.609838] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2910.109493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2910.966649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2917.541943] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 2925.568652] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2980.427529] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 17:37:14 (1788471434) [ 2982.311354] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2984.506634] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 17:37:18 (1788471438) [ 2989.537423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2989.542702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2989.553088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2993.120069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2993.122743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2995.708829] Lustre: server umount lustre-MDT0000 complete [ 2998.245716] LustreError: 94055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2998.259735] LustreError: 94055:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2999.841471] LustreError: 97326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471455 with bad export cookie 15651762253118930068 [ 2999.844289] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2999.865434] LustreError: 97326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3000.366938] Lustre: server umount lustre-MDT0001 complete [ 3009.568567] Lustre: 97670:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3014.625333] Lustre: 97673:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3016.062209] Lustre: server umount lustre-OST0000 complete [ 3028.949835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.647617] LustreError: 98373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3029.851856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3034.100696] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3035.111159] LustreError: 98374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3039.207976] LustreError: 98373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3043.625639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3044.543115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3049.570316] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3057.847959] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3065.087611] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3080.293847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3080.533110] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3080.539271] LustreError: 98363:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3080.563468] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3080.570311] LustreError: 100456:0:(tgt_mount.c:2734:server_fill_super()) Unable to start targets: -98 [ 3080.582401] LustreError: 100456:0:(tgt_mount.c:2192:server_put_super()) no obd lustre-OST0000 [ 3080.594555] LustreError: 100456:0:(tgt_mount.c:130:server_deregister_mount()) lustre-OST0000 not registered [ 3080.696473] Lustre: server umount lustre-OST0000 complete [ 3080.706047] LustreError: 100456:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3090.316524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3090.614024] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 3090.625409] LustreError: 98360:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 3090.697314] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 3090.719242] LustreError: 100926:0:(tgt_mount.c:2734:server_fill_super()) Unable to start targets: -98 [ 3090.740390] LustreError: 100926:0:(tgt_mount.c:2192:server_put_super()) no obd lustre-OST0000 [ 3090.748419] LustreError: 100926:0:(tgt_mount.c:130:server_deregister_mount()) lustre-OST0000 not registered [ 3090.823479] Lustre: server umount lustre-OST0000 complete [ 3090.825956] LustreError: 100926:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3102.314035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3102.525953] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3102.530037] Lustre: Found index 0 for lustre-OST0000, updating log [ 3102.535181] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3102.708471] Lustre: lustre-OST0000: new disk, initializing [ 3102.713179] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3102.717415] Lustre: 101461:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3103.064883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3109.884553] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3113.963021] Lustre: Failing over lustre-OST0000 [ 3114.264880] Lustre: server umount lustre-OST0000 complete [ 3118.565777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3118.568579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3118.577098] Lustre: Skipped 2 previous similar messages [ 3118.591770] Lustre: Skipped 3 previous similar messages [ 3123.684527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3124.899566] Lustre: server umount lustre-MDT0000 complete [ 3128.584887] LustreError: 98353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471584 with bad export cookie 15651762253118932854 [ 3128.591442] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3128.597346] LustreError: 98353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3139.305912] Lustre: server umount lustre-MDT0001 complete [ 3149.874369] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 17:40:03 (1788471603) [ 3169.049356] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3182.118642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3182.539310] LustreError: 103808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3182.735723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3189.242955] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3201.222917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3207.584788] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3211.846180] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3219.930333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3220.720742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3220.731972] Lustre: Skipped 1 previous similar message [ 3220.745983] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3226.859391] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 3226.899551] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 3226.900154] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3228.180588] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3236.875265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3236.961579] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3237.246165] Lustre: lustre-OST0001: new disk, initializing [ 3237.258964] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3237.276870] Lustre: 106128:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3244.803779] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3247.337095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3247.344734] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3247.404130] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3254.542989] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3261.872941] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3268.067374] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3268.071951] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3268.119201] Lustre: Skipped 1 previous similar message [ 3268.133924] Lustre: Skipped 1 previous similar message [ 3273.205570] LustreError: 104918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3273.254457] LustreError: 104918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3273.311551] Lustre: server umount lustre-MDT0001 complete [ 3285.500842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.576123] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3285.580814] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3285.659145] Lustre: 103797:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3286.564239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3286.567293] Lustre: Skipped 1 previous similar message [ 3286.972487] LustreError: 107667:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3286.993954] LustreError: 107667:0:(tgt_mount.c:739:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3287.005955] LustreError: 103801:0:(mgc_request.c:717:do_requeue()) failed processing log: -17 [ 3287.255715] LustreError: 107677:0:(obd_config.c:2100:class_config_llog_handler()) MGC192.168.201.123@tcp: cfg command failed: rc = -17 [ 3287.272403] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3287.272403] [ 3291.639140] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3291.750350] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3303.140515] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 17:42:37 (1788471757) [ 3322.832535] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3340.708727] Lustre: 109745:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3362.044922] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3366.023525] Lustre: 110880:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3460.076331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3460.088334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3460.103605] Lustre: Skipped 1 previous similar message [ 3460.120101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3460.127695] Lustre: Skipped 3 previous similar messages [ 3464.535791] Lustre: server umount lustre-MDT0000 complete [ 3465.697861] LustreError: 103803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3465.719581] LustreError: 103803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 3468.971461] LustreError: 103791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788471924 with bad export cookie 15651762253118933911 [ 3468.976914] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3468.983027] LustreError: 103791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3469.029216] Lustre: 111406:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3469.479565] Lustre: server umount lustre-MDT0001 complete [ 3475.936344] Lustre: 111614:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3475.957150] Lustre: 111614:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [ 3481.059475] Lustre: 111613:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3484.323696] Lustre: server umount lustre-OST0000 complete [ 3487.204456] Lustre: 93373:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788471926/real 1788471926] req@ffffa081aee5c700 x1875348092755456/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788471942 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3487.220655] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3487.231881] Lustre: Skipped 3 previous similar messages [ 3488.331145] Lustre: server umount lustre-OST0001 complete [ 3493.826305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3505.040626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3514.882572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3524.459388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3547.333160] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3559.269328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.287242] Lustre: lustre-MDT0000: reset Object Index mappings [ 3559.718222] LustreError: 114844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3559.797938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3564.899157] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3574.301144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3574.337904] Lustre: lustre-MDT0001: reset Object Index mappings [ 3579.689874] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3582.896493] Lustre: 115993:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3589.677968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3597.166217] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3599.705414] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:226 to 0x280000400:257) [ 3604.862255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3607.227909] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3613.201254] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3613.214287] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3616.556895] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3625.506977] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3630.226194] Lustre: 117871:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3682.788848] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 17:48:56 (1788472136) [ 3707.877146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3707.886549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3707.904646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3707.913641] Lustre: Skipped 3 previous similar messages [ 3714.128946] Lustre: server umount lustre-MDT0000 complete [ 3715.555754] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3715.557379] LustreError: 114844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3715.572465] Lustre: 118298:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3715.590406] LustreError: 114844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3715.608849] Lustre: 118298:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 2 previous similar messages [ 3725.210832] Lustre: server umount lustre-MDT0001 complete [ 3736.039566] Lustre: 118297:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3736.071704] Lustre: 118297:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 3 previous similar messages [ 3744.223274] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3744.593625] Lustre: server umount lustre-OST0000 complete [ 3755.422183] Lustre: server umount lustre-OST0001 complete [ 3764.576131] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 3778.109236] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3820.122642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.495293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3820.548065] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3820.658340] Lustre: lustre-MDT0000: new disk, initializing [ 3820.743743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3820.752907] Lustre: Skipped 3 previous similar messages [ 3820.767806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3826.973616] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3835.195541] Lustre: server umount lustre-MDT0000 complete [ 3846.011731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3846.470524] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3846.488499] Lustre: 122149:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3846.496876] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3852.292295] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3858.056217] Lustre: server umount lustre-MDT0000 complete [ 3869.318549] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 17:52:03 (1788472323) [ 3871.610965] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3873.951705] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 17:52:07 (1788472327) [ 3897.267935] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 3909.504324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.684161] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3926.903741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3927.069693] Lustre: 124212:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3927.174114] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3927.180753] Lustre: Skipped 1 previous similar message [ 3927.337837] Lustre: lustre-MDT0001: new disk, initializing [ 3927.519018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3927.539842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3933.197516] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3939.104852] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3946.147588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3946.466218] Lustre: lustre-OST0000: new disk, initializing [ 3946.470769] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3946.477240] Lustre: 125791:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3948.082660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3948.103205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3948.283526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3954.929899] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3968.969267] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3969.217121] Lustre: lustre-OST0001: new disk, initializing [ 3969.220827] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3969.235684] Lustre: 126664:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3971.077797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3971.098709] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3971.338717] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3976.723376] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 3987.827347] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3992.196754] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4385.634814] Lustre: 133083:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4385.643375] Lustre: 133083:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4385.658125] Lustre: 133083:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4385.679127] Lustre: 133083:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4385.690250] Lustre: 133083:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4385.700313] Lustre: 133083:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4684.648325] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 18:05:38 (1788473138) [ 4700.643802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4700.656298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4700.668820] Lustre: Skipped 5 previous similar messages [ 4700.673996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4700.681176] Lustre: Skipped 6 previous similar messages [ 4707.150319] Lustre: server umount lustre-MDT0000 complete [ 4708.837991] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4708.841348] LustreError: 124221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4708.861628] Lustre: 133414:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4708.896582] LustreError: 124221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4708.929726] Lustre: 133414:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 3 previous similar messages [ 4713.956346] Lustre: 133413:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4713.960955] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4713.989603] Lustre: Skipped 3 previous similar messages [ 4713.995810] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4714.001501] Lustre: Skipped 5 previous similar messages [ 4717.618233] Lustre: server umount lustre-MDT0001 complete [ 4724.200255] Lustre: 133414:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4724.231224] Lustre: 133414:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [ 4729.126221] Lustre: server umount lustre-OST0000 complete [ 4743.648832] Lustre: 133410:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4743.656521] Lustre: 133410:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 3 previous similar messages [ 4748.255184] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4748.843676] Lustre: server umount lustre-OST0001 complete [ 4756.680460] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 4771.013486] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 4815.396067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4815.597466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4815.623802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4815.690289] Lustre: lustre-MDT0000: new disk, initializing [ 4815.746755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4815.753454] Lustre: Skipped 5 previous similar messages [ 4815.766550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4820.661666] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4832.976445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4833.096934] Lustre: 136162:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4833.222893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4833.228176] Lustre: Skipped 1 previous similar message [ 4833.362806] Lustre: lustre-MDT0001: new disk, initializing [ 4833.545769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4833.563908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4839.502150] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4851.247323] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4857.591341] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4866.202936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4866.603891] Lustre: lustre-OST0000: new disk, initializing [ 4866.609919] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4866.629243] Lustre: 138130:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4867.978177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4867.996390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4868.160378] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4875.546158] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 4886.417942] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4898.738291] Lustre: lustre-MDD0000: changelog on [ 4901.941824] Lustre: lustre-MDD0001: changelog on [ 6975.580264] Lustre: lustre-MDD0001: changelog off [ 6977.483216] Lustre: 136168:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6978.524213] Lustre: lustre-MDD0000: changelog off [ 6987.744294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6987.747848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6987.758829] LustreError: Skipped 1 previous similar message [ 6987.763612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6987.791439] Lustre: Skipped 2 previous similar messages [ 6987.823879] Lustre: Skipped 1 previous similar message [ 6990.817672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6990.827962] Lustre: Skipped 1 previous similar message [ 6992.420447] Lustre: server umount lustre-OST0000 complete [ 6999.520810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6999.533677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6999.547293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7003.645255] Lustre: server umount lustre-MDT0000 complete [ 7006.179062] LustreError: 136174:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7006.184474] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7006.202204] LustreError: 136174:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7006.241580] Lustre: 146605:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7010.242610] Lustre: server umount lustre-MDT0001 complete [ 7025.304510] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 7028.038711] Key type lgssc unregistered [ 7028.377751] LNet: 147305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7028.382354] LNetError: 147305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7028.396705] LNet: Removed LNI 192.168.201.123@tcp [ 7029.324772] Key type .llcrypt unregistered [ 7029.349373] Key type ._llcrypt unregistered [ 7053.954208] Key type ._llcrypt registered [ 7053.955872] Key type .llcrypt registered [ 7054.138637] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 7072.834387] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 7073.703936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7073.818632] alg: No test for adler32 (adler32-zlib) [ 7075.426083] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 7075.777317] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 7077.639173] Key type lgssc registered [ 7079.772861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7131.407401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7131.446666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7132.618855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7132.635437] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7132.699978] Lustre: lustre-MDT0000: new disk, initializing [ 7132.826958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7132.872071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7139.290271] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7154.250206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7154.429514] Lustre: 150503:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7154.538815] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7154.550725] Lustre: Skipped 1 previous similar message [ 7154.683112] Lustre: lustre-MDT0001: new disk, initializing [ 7154.779378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7154.860844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7154.867882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7161.498918] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7177.014535] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7187.367664] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7197.940669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7198.265635] Lustre: lustre-OST0000: new disk, initializing [ 7198.269531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7198.274676] Lustre: 152476:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7198.400527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7205.094856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7205.113108] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7205.288905] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7207.192792] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7218.662525] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7225.377910] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_module obdecho/obdecho [ 7228.191915] LustreError: 153481:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7228.197673] LustreError: 153481:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7230.424069] LustreError: 153578:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7246.304549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7246.307706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7246.343926] Lustre: Skipped 1 previous similar message [ 7246.368267] Lustre: Skipped 2 previous similar messages [ 7251.431663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7251.438460] Lustre: Skipped 2 previous similar messages [ 7251.782922] Lustre: server umount lustre-MDT0000 complete [ 7256.519880] LustreError: 150496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475712 with bad export cookie 2572224405459994838 [ 7256.525165] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7256.529075] LustreError: 150496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7256.562896] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7256.590962] Lustre: Skipped 1 previous similar message [ 7256.606163] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7256.944482] Lustre: server umount lustre-MDT0001 complete [ 7262.189179] Lustre: server umount lustre-OST0000 complete [ 7290.083072] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 7308.843314] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 7359.327733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7359.719837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7359.784914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7359.938178] Lustre: lustre-MDT0000: new disk, initializing [ 7360.082188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7360.115350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7366.280380] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7378.686458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7378.919408] Lustre: 157325:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7379.089828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7379.106248] Lustre: Skipped 1 previous similar message [ 7379.291219] Lustre: lustre-MDT0001: new disk, initializing [ 7379.461068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7379.519384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7379.539586] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7385.287471] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7397.385724] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7404.412436] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7412.312265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7412.619554] Lustre: lustre-OST0000: new disk, initializing [ 7412.625463] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7412.642854] Lustre: 159293:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7412.756181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7418.055554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7418.068538] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7418.218874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7420.831509] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7433.056568] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7439.288942] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7439.739692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7445.219742] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7445.587707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7449.056686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7449.077259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7449.125992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7449.130061] Lustre: Skipped 1 previous similar message [ 7453.992691] Lustre: server umount lustre-OST0000 complete [ 7459.298453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7459.300315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7459.329952] Lustre: Skipped 2 previous similar messages [ 7459.350302] Lustre: Skipped 1 previous similar message [ 7466.797359] Lustre: server umount lustre-MDT0000 complete [ 7469.537713] LustreError: 157336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7469.553927] LustreError: 157336:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7470.579323] LustreError: 157316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788475926 with bad export cookie 2572224405459995825 [ 7470.587469] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7470.589777] LustreError: 157316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7471.081093] Lustre: server umount lustre-MDT0001 complete [ 7475.085114] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 18:52:09 (1788475929) [ 7477.153271] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7479.227671] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 18:52:13 (1788475933) [ 7487.726402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7488.462621] LustreError: 161706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7488.660646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7493.887511] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7494.123916] LustreError: 161707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7498.221219] LustreError: 161706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7503.329952] LustreError: 161707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7503.903813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7509.657165] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7518.316494] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7525.169927] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7533.047831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7533.436776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7533.439912] Lustre: Skipped 1 previous similar message [ 7541.164604] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7552.481123] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7560.169572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7560.175413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7560.193584] Lustre: Skipped 2 previous similar messages [ 7560.211307] Lustre: Skipped 4 previous similar messages [ 7565.232263] Lustre: server umount lustre-MDT0000 complete [ 7565.281358] LustreError: 161706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7565.305409] LustreError: 161706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7569.663334] LustreError: 161686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476025 with bad export cookie 2572224405459996721 [ 7569.677509] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7569.687076] LustreError: 161686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7570.408887] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7576.298134] Lustre: server umount lustre-MDT0001 complete [ 7579.553551] Lustre: 164495:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7583.905807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7584.741288] Lustre: 164496:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7589.860538] Lustre: 164496:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7594.976030] Lustre: 164496:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7598.789028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7599.979380] Lustre: 164497:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7600.005081] Lustre: 164497:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [ 7610.347121] Lustre: 164496:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7610.375601] Lustre: 164496:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 2 previous similar messages [ 7624.422028] LustreError: 165565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7624.442217] LustreError: 165565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7629.411702] Lustre: 164495:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7629.429251] Lustre: 164495:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 4 previous similar messages [ 7639.519713] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7645.360654] LustreError: MGC192.168.201.123@tcp: operation mgs_connect to node 0@lo failed: rc = -19 [ 7645.375685] LustreError: Skipped 1 previous similar message [ 7658.746900] LustreError: 165567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7658.800623] LustreError: 165567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [ 7665.120478] Lustre: 164497:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7665.130542] Lustre: 164497:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 7 previous similar messages [ 7685.455726] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7699.871411] Lustre: 165558:0:(mgc_request.c:2163:mgc_process_log()) MGC192.168.201.123@tcp: IR log lustre-mdtir failed, not fatal: rc = -108 [ 7705.487113] LustreError: 161698:0:(mgc_request.c:717:do_requeue()) failed processing log: -108 [ 7725.025350] LustreError: 165566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7725.061229] LustreError: 165566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 46 previous similar messages [ 7730.164568] Lustre: 164495:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7730.199084] Lustre: 164495:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 16 previous similar messages [ 7733.279455] Lustre: 161698:0:(mgc_request.c:2163:mgc_process_log()) MGC192.168.201.123@tcp: IR log lustre-mdtir failed, not fatal: rc = -108 [ 7740.029593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7745.271932] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7745.522450] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7756.265038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7761.738701] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7761.911072] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7770.910066] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7772.868799] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7779.897665] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7781.728950] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7802.850319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7802.854195] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7802.863146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7802.863155] Lustre: Skipped 4 previous similar messages [ 7802.909173] Lustre: Skipped 1 previous similar message [ 7803.204266] Lustre: server umount lustre-MDT0000 complete [ 7806.692877] LustreError: 164492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476262 with bad export cookie 2572224405460003385 [ 7806.694664] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7806.698455] LustreError: 164492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7807.107865] Lustre: server umount lustre-MDT0001 complete [ 7820.617094] Lustre: server umount lustre-OST0000 complete [ 7844.119750] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 7860.249499] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 7902.215695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7902.634224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7902.661774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7902.784640] Lustre: lustre-MDT0000: new disk, initializing [ 7902.929645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7902.941861] Lustre: Skipped 1 previous similar message [ 7902.970908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7907.509609] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7919.513619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7919.693078] Lustre: 170705:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7920.019847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7925.138704] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7937.029948] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7944.091826] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7953.121261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7953.642325] Lustre: lustre-OST0000: new disk, initializing [ 7953.646021] Lustre: Skipped 1 previous similar message [ 7953.649850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7953.657753] Lustre: Skipped 2 previous similar messages [ 7953.665924] Lustre: 172674:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7961.259766] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 7963.726376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7963.741164] Lustre: Skipped 1 previous similar message [ 7963.750740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7963.828241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7972.189560] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7978.102905] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7978.324387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7983.648333] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7983.926944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7989.732138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7989.734393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7989.750091] Lustre: Skipped 1 previous similar message [ 7989.756338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7989.788809] Lustre: Skipped 3 previous similar messages [ 7992.454738] Lustre: server umount lustre-OST0000 complete [ 7996.899162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8001.994971] Lustre: server umount lustre-MDT0000 complete [ 8005.089894] LustreError: 170742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8005.100949] LustreError: 170742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 8005.531075] LustreError: 170698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476461 with bad export cookie 2572224405460020262 [ 8005.549923] LustreError: 170698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8005.551818] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8006.438602] Lustre: server umount lustre-MDT0001 complete [ 8017.284651] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 19:01:10 (1788476470) [ 8024.317431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8029.738834] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8039.897215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8045.663496] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8055.277682] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8062.209276] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8069.583990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8077.831251] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8086.179340] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8096.867117] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 19:02:30 (1788476550) [ 8100.870045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8100.876799] Lustre: Skipped 3 previous similar messages [ 8100.880715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8100.883961] Lustre: Skipped 3 previous similar messages [ 8105.965066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8107.128465] Lustre: server umount lustre-OST0000 complete [ 8117.880505] Lustre: server umount lustre-MDT0000 complete [ 8121.760129] LustreError: 175080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476577 with bad export cookie 2572224405460021158 [ 8121.761606] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8121.799501] LustreError: 175080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8122.070436] Lustre: server umount lustre-MDT0001 complete [ 8135.002947] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 8138.785694] Key type lgssc unregistered [ 8139.175056] LNet: 178830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8139.185451] LNetError: 178830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8139.215652] LNet: Removed LNI 192.168.201.123@tcp [ 8140.210533] Key type .llcrypt unregistered [ 8140.212586] Key type ._llcrypt unregistered [ 8162.221173] Key type ._llcrypt registered [ 8162.228495] Key type .llcrypt registered [ 8162.410509] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 8163.825917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8163.851789] alg: No test for adler32 (adler32-zlib) [ 8164.814357] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 8165.010570] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 8166.655185] Key type lgssc registered [ 8167.665209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8176.802856] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8176.839158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8177.897996] Lustre: 180082:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 8193.311388] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8198.431312] LustreError: 180069:0:(mgc_request.c:317:do_config_log_add()) MGC192.168.201.123@tcp: failed processing log, type 4: rc = -110 [ 8203.233289] Lustre: 180084:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 8203.551972] LustreError: 180069:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8203.586573] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8203.609877] LustreError: MGC192.168.201.123@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8203.630188] LustreError: 180069:0:(tgt_mount.c:1948:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8203.651822] LustreError: 180069:0:(tgt_mount.c:2734:server_fill_super()) Unable to start targets: -22 [ 8203.665785] LustreError: 180069:0:(tgt_mount.c:2192:server_put_super()) no obd lustre-OST0000 [ 8203.830530] Lustre: server umount lustre-OST0000 complete [ 8203.833417] LustreError: 180069:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8216.747362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8217.221240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8223.304831] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8226.789032] LustreError: 180575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8226.818811] LustreError: 180575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8231.907082] LustreError: 180576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8234.356933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8235.041275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8239.948820] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8247.523817] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8254.247389] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8261.380587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8261.602859] LustreError: 182397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8261.679127] LustreError: 182392:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8261.688138] Lustre: 182392:0:(mgc_request_server.c:733:mgc_llog_local_copy()) MGC192.168.201.123@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8261.798893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8268.752180] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8277.218294] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8281.089121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8281.100276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8281.117743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8284.647175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8284.649458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8289.772412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8289.786751] Lustre: Skipped 4 previous similar messages [ 8294.368758] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8294.562301] Lustre: server umount lustre-MDT0000 complete [ 8294.879846] LustreError: 180576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8294.893150] LustreError: 180576:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8298.316528] LustreError: 180555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476754 with bad export cookie 220698626115347081 [ 8298.335147] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8298.803731] Lustre: server umount lustre-MDT0001 complete [ 8309.156216] Lustre: 183557:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 8312.380524] Lustre: server umount lustre-OST0000 complete [ 8322.352588] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 8333.046875] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 8370.340102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8370.621194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8370.668744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8370.767282] Lustre: lustre-MDT0000: new disk, initializing [ 8370.860318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8370.889253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8376.260349] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8387.963387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8388.077797] Lustre: 185923:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8388.179144] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8388.185023] Lustre: Skipped 1 previous similar message [ 8388.278450] Lustre: lustre-MDT0001: new disk, initializing [ 8388.392253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8388.426688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8388.434538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8393.209985] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8404.203484] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8411.855272] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8420.592273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8420.825765] Lustre: lustre-OST0000: new disk, initializing [ 8420.828490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8420.833229] Lustre: 187889:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8420.907844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8426.086183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8426.101986] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8426.257494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8428.844810] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8443.247359] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8448.809982] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8449.104427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8454.446958] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8454.926586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8462.307054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8462.310958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8462.314685] LustreError: Skipped 1 previous similar message [ 8462.316455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8462.332479] Lustre: Skipped 2 previous similar messages [ 8463.668552] Lustre: server umount lustre-OST0000 complete [ 8470.501634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8470.512854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8470.527117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8470.534032] Lustre: Skipped 1 previous similar message [ 8474.445644] Lustre: server umount lustre-MDT0000 complete [ 8477.667117] LustreError: 185933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8477.692903] LustreError: 185933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8478.877093] LustreError: 189228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788476934 with bad export cookie 220698626115347844 [ 8478.878902] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8478.885475] LustreError: 189228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8479.469197] Lustre: server umount lustre-MDT0001 complete [ 8489.901361] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 19:09:03 (1788476943) [ 8506.652965] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 8509.367854] Key type lgssc unregistered [ 8509.647766] LNet: 190835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8509.667807] LNetError: 190835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8509.688872] LNet: Removed LNI 192.168.201.123@tcp [ 8510.684262] Key type .llcrypt unregistered [ 8510.686678] Key type ._llcrypt unregistered [ 8529.801527] Key type ._llcrypt registered [ 8529.805866] Key type .llcrypt registered [ 8529.946438] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 8530.834281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8530.872884] alg: No test for adler32 (adler32-zlib) [ 8531.933641] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 8532.147233] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 8533.855186] Key type lgssc registered [ 8535.370268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8544.197542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8544.223734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8545.895132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8550.976037] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8555.496093] LustreError: 192052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8555.515762] LustreError: 192052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8560.582899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8560.611791] LustreError: 192053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8561.064462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8566.299402] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8577.043713] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8584.186281] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8593.609377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8593.800796] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8593.805234] LustreError: 193917:0:(llog.c:1696:llog_backup()) MGC192.168.201.123@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8593.813143] Lustre: 193917:0:(mgc_request_server.c:730:mgc_llog_local_copy()) MGC192.168.201.123@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8593.939385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8600.410623] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8609.487617] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8612.321196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8612.329158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8612.344728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8615.394041] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8615.396445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8615.408523] Lustre: Skipped 1 previous similar message [ 8615.418637] Lustre: Skipped 2 previous similar messages [ 8618.555754] Lustre: server umount lustre-MDT0000 complete [ 8620.517229] LustreError: 193036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8620.538152] LustreError: 193036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8622.854843] LustreError: 192031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477078 with bad export cookie 13344960878319794351 [ 8622.866756] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8622.872781] LustreError: 192031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8623.325568] Lustre: server umount lustre-MDT0001 complete [ 8632.483165] Lustre: 195069:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 8637.794765] Lustre: server umount lustre-OST0000 complete [ 8647.519440] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 8659.200796] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 8703.929930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8704.115311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8704.142546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8704.211524] Lustre: lustre-MDT0000: new disk, initializing [ 8704.287901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8704.319166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8709.088831] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8720.963773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8721.053906] Lustre: 197451:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8721.171594] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8721.177042] Lustre: Skipped 1 previous similar message [ 8721.188694] LustreError: 198233:0:(llog.c:1696:llog_backup()) MGC192.168.201.123@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8721.197068] LustreError: 198233:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8721.203207] Lustre: 198233:0:(mgc_request_server.c:753:mgc_llog_local_copy()) MGC192.168.201.123@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8721.224078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8721.238372] Lustre: Skipped 1 previous similar message [ 8721.313787] Lustre: lustre-MDT0001: new disk, initializing [ 8721.399989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8721.439471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8721.453572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8726.116843] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8736.681212] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8742.611261] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8749.555706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8749.844509] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8749.850407] Lustre: Skipped 1 previous similar message [ 8749.852330] LustreError: 199394:0:(llog.c:1696:llog_backup()) MGC192.168.201.123@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8749.868257] LustreError: 199394:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8749.877240] Lustre: 199394:0:(mgc_request_server.c:753:mgc_llog_local_copy()) MGC192.168.201.123@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8749.886019] Lustre: 199394:0:(mgc_request_server.c:753:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8749.904532] Lustre: lustre-OST0000: new disk, initializing [ 8749.909438] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8749.920364] Lustre: 199417:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8750.037240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8755.687187] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8755.692366] Lustre: Skipped 1 previous similar message [ 8755.695225] LustreError: 197454:0:(llog.c:1696:llog_backup()) MGC192.168.201.123@tcp: failed to backup log lustre-client: rc = -28 [ 8755.701328] LustreError: 197454:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8755.708738] Lustre: 197454:0:(mgc_request_server.c:753:mgc_llog_local_copy()) MGC192.168.201.123@tcp: failed to copy new config lustre-client: rc = -28 [ 8755.726158] Lustre: 197454:0:(mgc_request_server.c:753:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8755.823027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8755.836786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8756.010710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8756.295320] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8767.268543] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8773.555719] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8773.880416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8779.526571] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8779.851390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8786.918803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8786.922347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8786.929363] LustreError: Skipped 1 previous similar message [ 8786.930470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8786.937974] Lustre: Skipped 1 previous similar message [ 8786.948648] Lustre: Skipped 1 previous similar message [ 8788.249630] Lustre: server umount lustre-OST0000 complete [ 8793.057984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8793.068816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8793.092994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8797.153872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8797.156474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8798.533898] Lustre: server umount lustre-MDT0000 complete [ 8802.060064] LustreError: 197441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477257 with bad export cookie 13344960878319795107 [ 8802.067949] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8802.076956] LustreError: 197441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8802.385404] Lustre: server umount lustre-MDT0001 complete [ 8811.244243] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 19:14:25 (1788477265) [ 8827.758485] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 8837.552477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8837.843175] LustreError: 202545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8837.884020] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8841.620655] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8843.232547] LustreError: 202546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8850.434590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8855.386112] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8858.830726] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8865.779540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8873.226879] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8882.526192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8882.776499] Lustre: lustre-OST0001: new disk, initializing [ 8882.780606] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8882.791558] Lustre: 204857:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8882.883310] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8882.887348] Lustre: Skipped 2 previous similar messages [ 8888.510648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8888.519090] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8888.611305] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8889.824669] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8901.913832] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8914.451802] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8916.738870] Lustre: DEBUG MARKER: ADD OST3 [ 8922.831870] loop: module loaded [ 8922.960551] loop0: detected capacity change from 0 to 204800000 [ 8923.224837] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8923.236579] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8923.255432] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8930.311909] loop0: detected capacity change from 0 to 204800000 [ 8931.033563] Lustre: DEBUG MARKER: STOP OST3 [ 8937.013333] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8937.355566] Lustre: lustre-OST0003: new disk, initializing [ 8937.361525] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8937.366233] Lustre: 206855:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8938.712943] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8938.828972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8938.839445] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8938.894214] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8940.768133] LustreError: 206947:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8942.191641] Lustre: Failing over lustre-OST0003 [ 8942.349096] Lustre: server umount lustre-OST0003 complete [ 8944.111866] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8944.113821] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8944.125385] LustreError: 205393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8944.125396] LustreError: 205393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8944.131025] Lustre: Skipped 1 previous similar message [ 8945.154460] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8955.727554] Lustre: DEBUG MARKER: START OST3 again [ 8964.446530] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8964.587147] LustreError: 205393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8964.611636] LustreError: 205393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8964.777849] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8964.782452] Lustre: Skipped 1 previous similar message [ 8964.796600] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8964.873662] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8966.469966] Lustre: lustre-OST0003: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 8966.469991] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8971.081193] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 8992.799385] Lustre: Failing over lustre-OST0003 [ 8992.889481] Lustre: server umount lustre-OST0003 complete [ 8995.833886] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9000.937511] LustreError: 204032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9000.952853] LustreError: 204032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9011.192306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9011.201588] Lustre: Skipped 3 previous similar messages [ 9017.100071] Lustre: server umount lustre-MDT0000 complete [ 9021.070664] LustreError: 202526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477476 with bad export cookie 13344960878319796003 [ 9021.081107] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9021.082534] LustreError: 202526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9021.116375] Lustre: 209560:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9026.530978] Lustre: 209563:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9027.826314] Lustre: server umount lustre-MDT0001 complete [ 9031.650085] Lustre: 209560:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9032.496589] Lustre: server umount lustre-OST0000 complete [ 9036.342975] Lustre: server umount lustre-OST0001 complete [ 9044.141715] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 9053.433131] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 9097.931511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9098.196343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9098.230874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9098.393632] Lustre: lustre-MDT0000: new disk, initializing [ 9098.481730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9098.511952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9102.689783] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 9114.553699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9114.675250] Lustre: 212322:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9114.748948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9114.752349] Lustre: Skipped 1 previous similar message [ 9114.846892] Lustre: lustre-MDT0001: new disk, initializing [ 9115.005117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9115.013717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9119.686830] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 9130.220830] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9137.301539] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9145.950444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9146.556513] Lustre: 214293:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9147.692122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9147.700147] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9147.822122] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9153.915411] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 9164.108555] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9169.671196] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9169.954314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9175.486237] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9175.673404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9178.595120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9178.595213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9178.602628] LustreError: Skipped 1 previous similar message [ 9178.613729] Lustre: Skipped 7 previous similar messages [ 9178.620846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9178.620866] Lustre: Skipped 8 previous similar messages [ 9184.573895] Lustre: server umount lustre-OST0000 complete [ 9191.904445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9195.617169] Lustre: server umount lustre-MDT0000 complete [ 9199.082097] LustreError: 212329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9199.085702] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9199.114537] LustreError: 212329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 9199.152512] Lustre: 215915:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9200.378754] Lustre: server umount lustre-MDT0001 complete [ 9204.780194] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 19:20:58 (1788477658) [ 9223.823675] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 9234.476992] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 9273.065523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9273.273139] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9273.307070] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9273.313588] Lustre: Skipped 1 previous similar message [ 9273.401825] Lustre: lustre-MDT0000: new disk, initializing [ 9273.405637] Lustre: Skipped 1 previous similar message [ 9273.554466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9277.758802] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 9287.944756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9288.090222] Lustre: 218906:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9288.446627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9292.705552] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 9302.540185] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9308.644771] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9315.471392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9315.832555] Lustre: 220878:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9317.046235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9322.152665] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 9332.371690] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9337.012416] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9337.221083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9343.280502] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9343.494318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9347.555624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9347.588807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9347.629350] Lustre: Skipped 2 previous similar messages [ 9347.642166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9347.646702] Lustre: Skipped 6 previous similar messages [ 9352.077638] Lustre: server umount lustre-OST0000 complete [ 9371.103136] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9371.289890] Lustre: server umount lustre-MDT0000 complete [ 9373.667046] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9373.668761] LustreError: 218918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9373.700302] Lustre: 222498:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9373.713390] LustreError: 218918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9375.647954] Lustre: server umount lustre-MDT0001 complete [ 9385.525344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9386.023143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9386.027348] Lustre: Skipped 5 previous similar messages [ 9390.319791] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 9399.601444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9404.279995] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 9412.094753] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9417.498734] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9424.489487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9430.797888] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 9438.222425] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9449.447045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9449.454386] LustreError: Skipped 1 previous similar message [ 9449.455415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9449.475702] Lustre: Skipped 9 previous similar messages [ 9450.620571] Lustre: server umount lustre-OST0000 complete [ 9460.378477] Lustre: server umount lustre-MDT0000 complete [ 9464.137151] LustreError: 223082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788477919 with bad export cookie 13344960878319800287 [ 9464.142455] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9464.152269] LustreError: 223082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9464.183362] Lustre: 226055:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9464.524492] Lustre: server umount lustre-MDT0001 complete [ 9578.865909] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [ 9581.256326] Key type lgssc unregistered [ 9581.516896] LNet: 226771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9581.522278] LNetError: 226771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9581.547984] LNet: Removed LNI 192.168.201.123@tcp [ 9582.234209] Key type .llcrypt unregistered [ 9582.242825] Key type ._llcrypt unregistered [ 9605.001818] Key type ._llcrypt registered [ 9605.003231] Key type .llcrypt registered [ 9605.094728] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 9618.858721] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 9619.824782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9619.998929] alg: No test for adler32 (adler32-zlib) [ 9621.305901] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [ 9621.541700] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [ 9623.215451] Key type lgssc registered [ 9624.508445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9662.118931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9662.144215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9663.401761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9663.443941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9663.564366] Lustre: lustre-MDT0000: new disk, initializing [ 9663.636734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9663.666424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9668.359516] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 9680.038538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9680.139695] Lustre: 230126:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9680.250604] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9680.255129] Lustre: Skipped 1 previous similar message [ 9680.367222] Lustre: lustre-MDT0001: new disk, initializing [ 9680.494261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9680.550721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9680.572464] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9687.366303] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 9698.117916] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9703.967735] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9710.685602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9710.898192] Lustre: lustre-OST0000: new disk, initializing [ 9710.902581] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9710.908970] Lustre: 232090:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9711.012254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9718.181948] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 9720.520930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9720.531891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9720.647319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9728.728231] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9734.291490] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9734.489203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9739.348958] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9739.612471] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9746.400895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9746.407421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9746.416964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9747.578965] Lustre: server umount lustre-OST0000 complete [ 9751.521916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9751.536549] Lustre: Skipped 1 previous similar message [ 9751.542098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9751.552166] Lustre: Skipped 2 previous similar messages [ 9756.646569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9756.656147] Lustre: Skipped 1 previous similar message [ 9756.894727] Lustre: server umount lustre-MDT0000 complete [ 9760.707847] LustreError: 230116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478216 with bad export cookie 734322180443335964 [ 9760.718254] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9760.936600] Lustre: server umount lustre-MDT0001 complete [ 9781.006240] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 9791.591748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9791.883027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9796.291808] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 9797.099455] LustreError: 235025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9797.119912] LustreError: 235025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9801.188519] LustreError: 235026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9804.569616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9804.924808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9809.554400] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 9812.486420] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9819.162818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9819.568126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9827.130748] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 9836.480350] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9836.683168] Lustre: lustre-OST0001: new disk, initializing [ 9836.688649] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9836.696805] Lustre: 237333:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9836.836753] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9842.933687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9842.941785] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9843.153708] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9844.486952] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 9857.562236] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9866.529419] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9877.889289] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 19:32:12 (1788478332) [ 9881.568684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9881.571547] LustreError: Skipped 1 previous similar message [ 9881.582245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9881.597443] Lustre: Skipped 1 previous similar message [ 9881.603191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9884.132658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9884.165140] Lustre: Skipped 2 previous similar messages [ 9886.429154] Lustre: server umount lustre-MDT0000 complete [ 9889.286700] LustreError: 235026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9889.291939] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9889.336257] LustreError: 235026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9889.364486] Lustre: 238544:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9890.478125] Lustre: server umount lustre-MDT0001 complete [ 9894.372209] Lustre: 238542:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9899.493324] Lustre: 238543:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9904.608688] Lustre: 238544:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9904.808771] Lustre: server umount lustre-OST0000 complete [ 9909.733329] Lustre: 238543:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 9919.313695] Lustre: server umount lustre-OST0001 complete [ 9927.296903] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [ 9938.162908] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [ 9972.314398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9972.573745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9972.607735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9972.722308] Lustre: lustre-MDT0000: new disk, initializing [ 9972.829521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9972.848594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9977.064874] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [ 9986.786934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9986.874604] Lustre: 241286:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9986.969410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9986.975667] Lustre: Skipped 1 previous similar message [ 9987.052986] Lustre: lustre-MDT0001: new disk, initializing [ 9987.173076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9987.177878] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9991.627993] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [10001.613444] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10007.579655] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10015.449955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10015.686995] Lustre: lustre-OST0000: new disk, initializing [10015.690804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10015.695460] Lustre: 243260:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10017.394937] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10017.404877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10017.524127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10023.519381] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [10034.538759] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10039.758806] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10039.937973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10044.380734] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10044.691788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10048.487722] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10048.500380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10048.501233] Lustre: Skipped 1 previous similar message [10048.503237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10048.536893] Lustre: Skipped 5 previous similar messages [10058.721709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10058.733780] Lustre: Skipped 3 previous similar messages [10061.279211] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10061.455606] Lustre: server umount lustre-OST0000 complete [10068.962764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10068.976120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10071.512577] Lustre: server umount lustre-MDT0000 complete [10074.083392] LustreError: 241294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10074.088992] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10074.099896] LustreError: 241294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10074.102244] Lustre: 244859:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10074.102480] Lustre: 244859:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [10075.528995] Lustre: server umount lustre-MDT0001 complete [10083.865987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10084.199301] LustreError: 245482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10084.245506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10084.252560] Lustre: Skipped 2 previous similar messages [10087.965252] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [10094.570960] LustreError: 245483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10094.591643] LustreError: 245483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10095.919620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10100.781976] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [10108.045199] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10113.956159] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10121.542755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10127.960624] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [10134.778732] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10278.884086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10278.890524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10278.920777] Lustre: Skipped 1 previous similar message [10278.930521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10278.933438] Lustre: Skipped 2 previous similar messages [10284.603178] Lustre: server umount lustre-OST0000 complete [10295.027815] Lustre: server umount lustre-MDT0000 complete [10295.271984] LustreError: 245482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10295.298441] LustreError: 245482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10298.890095] LustreError: 245464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788478754 with bad export cookie 734322180443340017 [10298.894180] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10298.900285] LustreError: 245464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10298.919411] Lustre: 248382:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10299.248599] Lustre: server umount lustre-MDT0001 complete [10404.950510] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [10409.336614] Key type lgssc unregistered [10409.850924] LNet: 249093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10409.864275] LNetError: 249093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10410.920901] LNet: Removed LNI 192.168.201.123@tcp [10412.314670] Key type .llcrypt unregistered [10412.318637] Key type ._llcrypt unregistered [10440.985151] Key type ._llcrypt registered [10440.992167] Key type .llcrypt registered [10441.127209] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [10459.263635] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [10460.505237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10460.576872] alg: No test for adler32 (adler32-zlib) [10461.582686] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [10461.780675] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [10463.463273] Key type lgssc registered [10464.841783] Lustre: Echo OBD driver; http://www.lustre.org/ [10516.271718] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [10527.403531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10527.446074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10528.659592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10528.678484] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10528.733234] Lustre: lustre-MDT0000: new disk, initializing [10528.817759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10528.835512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10533.925763] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [10545.455478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10545.602932] Lustre: 253162:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10545.705470] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10545.712384] Lustre: Skipped 1 previous similar message [10545.810490] Lustre: lustre-MDT0001: new disk, initializing [10545.952506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10545.997420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10546.029095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10550.931201] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [10555.666550] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10564.688023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10565.147420] Lustre: lustre-OST0000: new disk, initializing [10565.155098] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10565.172965] Lustre: 254795:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10565.305385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10568.101918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10568.125792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10568.256760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10573.751138] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [10586.312823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10586.477898] Lustre: lustre-OST0001: new disk, initializing [10586.480868] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10586.485767] Lustre: 255668:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10586.651666] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10592.115294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10592.122992] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10592.277234] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10594.725438] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [10606.611096] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10618.501200] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10633.188783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10633.193869] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10633.211712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10633.230974] Lustre: Skipped 2 previous similar messages [10633.255729] Lustre: Skipped 3 previous similar messages [10638.307346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10639.449294] Lustre: server umount lustre-MDT0000 complete [10643.366697] LustreError: 254793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479099 with bad export cookie 8922130935845666454 [10643.373769] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10643.380581] LustreError: 254793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10643.394101] Lustre: 256820:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10643.425352] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10643.432594] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10643.433043] Lustre: Skipped 2 previous similar messages [10643.441497] Lustre: Skipped 4 previous similar messages [10643.858607] Lustre: server umount lustre-MDT0001 complete [10648.555509] Lustre: 256822:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10649.708563] Lustre: server umount lustre-OST0000 complete [10653.664061] Lustre: 256820:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10653.881740] Lustre: server umount lustre-OST0001 complete [10668.311273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10677.083027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10684.567830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10694.234802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10714.246490] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [10726.045895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10726.066533] Lustre: lustre-MDT0000: reset Object Index mappings [10726.696194] LustreError: 260682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10726.719440] LustreError: 260682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10726.810022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10732.004197] LustreError: 260683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10732.490171] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [10737.122789] LustreError: 260682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10742.249116] LustreError: 260683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10743.196784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10743.235494] Lustre: lustre-MDT0001: reset Object Index mappings [10743.811904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10749.017628] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [10752.308103] Lustre: 261825:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10759.041884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10765.909234] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [10768.571493] LustreError: 262178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10771.663168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10774.810434] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10775.177568] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10775.189539] Lustre: Skipped 1 previous similar message [10780.687033] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10781.742538] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [10789.845543] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10793.989361] Lustre: 263695:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10800.098177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10800.113488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10800.126391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10801.487151] Lustre: server umount lustre-MDT0000 complete [10805.408144] LustreError: 260664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479261 with bad export cookie 8922130935845673342 [10805.415732] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10805.422070] LustreError: 260664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10805.452788] Lustre: 263901:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10805.760042] Lustre: server umount lustre-MDT0001 complete [10811.360976] Lustre: 263901:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10811.373983] Lustre: 263901:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [10819.671467] Lustre: server umount lustre-OST0000 complete [10821.608601] Lustre: 263901:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [10821.622343] Lustre: 263901:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [10822.624333] Lustre: 250668:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788479262/real 1788479262] req@ffffa081bf166d80 x1875356069009792/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788479278 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10822.665333] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10822.679030] Lustre: Skipped 3 previous similar messages [10823.714162] Lustre: server umount lustre-OST0001 complete [10844.863887] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 19:48:19 (1788479299) [10864.974208] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [10880.553197] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [10925.546484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10925.783797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10925.810952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10925.877328] Lustre: lustre-MDT0000: new disk, initializing [10925.946311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10925.961627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10930.652675] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [10943.012929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10943.106586] Lustre: 267977:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10943.171255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10943.175716] Lustre: Skipped 1 previous similar message [10943.248524] Lustre: lustre-MDT0001: new disk, initializing [10943.423834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10943.437708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10948.571658] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [10958.401301] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10964.130949] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10971.581245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10971.965406] Lustre: lustre-OST0000: new disk, initializing [10971.974546] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10971.979910] Lustre: 269948:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10973.662022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10973.675850] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10973.832949] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10979.474206] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [10990.142811] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10995.219577] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10995.414171] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11000.794108] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11001.034095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11004.385251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11004.394573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11004.424962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11004.435721] Lustre: Skipped 4 previous similar messages [11009.190885] Lustre: server umount lustre-OST0000 complete [11015.145204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11015.147176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11015.163335] LustreError: Skipped 1 previous similar message [11015.185698] Lustre: Skipped 1 previous similar message [11015.187674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11015.202845] Lustre: Skipped 2 previous similar messages [11019.360746] Lustre: server umount lustre-MDT0000 complete [11020.257571] LustreError: 267990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11020.285881] LustreError: 267990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11023.187963] LustreError: 267971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479479 with bad export cookie 8922130935845675491 [11023.194231] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11023.202559] LustreError: 267971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11023.233676] Lustre: 271600:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [11023.250819] Lustre: 271600:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 1 previous similar message [11023.542620] Lustre: server umount lustre-MDT0001 complete [11044.095486] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [11055.757791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11055.954727] LustreError: 272885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11055.987240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11055.992494] Lustre: Skipped 2 previous similar messages [11061.807962] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11072.194359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11078.388391] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11082.309682] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11089.986360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11099.043705] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11110.493348] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11110.859140] Lustre: lustre-OST0001: new disk, initializing [11110.900653] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11110.917886] Lustre: 275197:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11112.953464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11112.973924] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11113.133512] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11118.883895] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11129.569927] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11133.659967] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11139.577899] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11139.702328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11151.948735] Lustre: server umount lustre-MDT0000 complete [11154.400960] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11154.407208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11154.426533] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11154.431658] Lustre: Skipped 1 previous similar message [11154.456207] LustreError: 273826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11154.485072] Lustre: 276801:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [11154.489481] LustreError: 273826:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11156.916446] Lustre: server umount lustre-MDT0001 complete [11171.990839] Lustre: server umount lustre-OST0000 complete [11175.910293] Lustre: 250667:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788479615/real 1788479615] req@ffffa080829dd500 x1875356069121408/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788479631 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11177.004668] Lustre: server umount lustre-OST0001 complete [11196.208909] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [11208.029734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11208.621195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11208.630726] Lustre: Skipped 3 previous similar messages [11213.789648] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11218.919305] LustreError: 278545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11218.957453] LustreError: 278545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11224.699763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11231.069651] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11234.336647] Lustre: 279685:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11242.179326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11249.997854] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11253.977220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11261.646870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11267.051181] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11268.218445] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11275.929701] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11279.916850] Lustre: 281564:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11285.496478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11285.585374] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11300.387838] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 19:55:54 (1788479754) [11317.217726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11317.222667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11317.248763] Lustre: Skipped 4 previous similar messages [11317.257558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11322.797129] Lustre: server umount lustre-MDT0000 complete [11327.414780] LustreError: 278525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788479783 with bad export cookie 8922130935845682890 [11327.439766] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11327.440248] LustreError: 278525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11327.466428] Lustre: 282599:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [11327.478719] Lustre: 282599:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 5 previous similar messages [11334.025943] Lustre: server umount lustre-MDT0001 complete [11345.304373] Lustre: server umount lustre-OST0000 complete [11349.421057] Lustre: server umount lustre-OST0001 complete [11358.138104] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [11370.435777] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [11418.859554] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [11429.982227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11430.247830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11430.310254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11430.401813] Lustre: lustre-MDT0000: new disk, initializing [11430.486891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11434.682488] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11445.344671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11445.476618] Lustre: 286096:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11445.744356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11450.517730] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11464.111719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11464.452697] Lustre: 287729:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11465.891080] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11471.186905] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11482.455281] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11482.727406] Lustre: 288597:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11482.982339] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11482.985788] Lustre: Skipped 6 previous similar messages [11484.320358] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11484.331026] Lustre: Skipped 1 previous similar message [11484.416108] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11489.570419] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11499.734598] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11503.471674] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11503.477340] Lustre: Skipped 1 previous similar message [11520.429570] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 19:59:34 (1788479974) [11523.537697] Lustre: server umount lustre-MDT0000 complete [11525.606025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11525.624154] Lustre: Skipped 5 previous similar messages [11525.631219] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11525.639288] LustreError: 286108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11525.659328] Lustre: 289846:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [11525.685964] LustreError: 286108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [11525.716510] Lustre: 289846:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 5 previous similar messages [11528.246433] Lustre: server umount lustre-MDT0001 complete [11543.296914] Lustre: server umount lustre-OST0000 complete [11547.103937] Lustre: 250668:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788479986/real 1788479986] req@ffffa0808b395f80 x1875356069281408/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788480002 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11548.248424] Lustre: server umount lustre-OST0001 complete [11558.197194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11563.974906] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11573.110955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11579.069298] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11586.891467] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11594.074160] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11601.443270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11608.643276] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11616.557879] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11637.324287] loop1: detected capacity change from 0 to 204800000 [11637.556152] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11637.583265] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11637.605273] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11643.042605] loop1: detected capacity change from 0 to 204800000 [11649.228403] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11649.281522] LustreError: 290840:0:(mgs_llog.c:680:mgs_find_or_make_fsdb_nolock()) fs5802: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11649.298568] LustreError: 290840:0:(mgs_llog.c:739:mgs_check_index()) Can't get db for fs5802 [11649.305986] LustreError: 290840:0:(mgs_handler.c:517:mgs_target_reg()) fs5802-MDT0001: Error checking target index: rc = -13 [11649.318966] LustreError: fs5802-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11649.330043] LustreError: 294583:0:(tgt_mount.c:2734:server_fill_super()) Unable to start targets: -13 [11649.344037] LustreError: 294583:0:(tgt_mount.c:2192:server_put_super()) no obd fs5802-MDT0001 [11649.349088] LustreError: 294583:0:(tgt_mount.c:130:server_deregister_mount()) fs5802-MDT0001 not registered [11649.376444] Lustre: server umount fs5802-MDT0001 complete [11649.380492] LustreError: 294583:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11658.194726] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11658.272886] Lustre: Setting parameter fs5802-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs5802-MDT0001 [11658.313560] Lustre: srv-fs5802-MDT0001: No data found on store. Initialize space. [11658.317778] Lustre: Skipped 4 previous similar messages [11658.449895] Lustre: fs5802-MDT0001: new disk, initializing [11658.457866] Lustre: Skipped 3 previous similar messages [11658.626049] Lustre: srv-fs5802-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11658.639386] Lustre: 295051:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5802-MDT0001-mdtlov: can't create new llog: rc = -115 [11658.646847] LustreError: 295051:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5802-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11663.509753] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11670.143516] Lustre: server umount fs5802-MDT0001 complete [11684.839761] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11685.079718] loop1: detected capacity change from 0 to 204800000 [11691.449580] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11691.932554] Lustre: 296618:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5802-MDT0001-mdtlov: can't create new llog: rc = -115 [11691.945380] LustreError: 296618:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5802-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11697.246471] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11705.768178] Lustre: server umount fs5802-MDT0001 complete [11719.966128] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.23@tcp (stopping) [11719.974825] Lustre: Skipped 8 previous similar messages [11720.677220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11720.696081] Lustre: Skipped 6 previous similar messages [11725.211168] Lustre: server umount lustre-MDT0000 complete [11730.930787] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11735.123484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11741.166025] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11741.613252] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11749.876537] loop1: detected capacity change from 0 to 204800000 [11755.150154] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11755.484594] Lustre: 299070:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs5802-MDT0001-mdtlov: can't create new llog: rc = -115 [11755.502751] LustreError: 299070:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs5802-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11760.163943] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11765.285848] Lustre: server umount fs5802-MDT0001 complete [11782.123460] LustreError: 292682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11782.146583] LustreError: 292682:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 109 previous similar messages [11792.373523] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11792.383370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11792.391910] LustreError: Skipped 2 previous similar messages [11792.407560] Lustre: Skipped 5 previous similar messages [11795.363737] Lustre: server umount lustre-OST0000 complete [11805.654933] Lustre: server umount lustre-MDT0000 complete [11807.717470] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11807.741361] Lustre: 300726:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [11807.760200] Lustre: 300726:0:(ldlm_lib.c:1248:target_handle_connect()) Skipped 6 previous similar messages [11811.397985] Lustre: server umount lustre-MDT0001 complete [11824.994840] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [11828.044761] Key type lgssc unregistered [11828.366808] LNet: 301418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11828.376281] LNetError: 301418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11829.422646] LNet: Removed LNI 192.168.201.123@tcp [11830.325252] Key type .llcrypt unregistered [11830.330321] Key type ._llcrypt unregistered [11851.778914] Key type ._llcrypt registered [11851.780956] Key type .llcrypt registered [11851.888262] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [11853.161357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11853.212820] alg: No test for adler32 (adler32-zlib) [11854.296619] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [11854.564408] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [11856.249614] Key type lgssc registered [11857.554297] Lustre: Echo OBD driver; http://www.lustre.org/ [11866.373990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11866.398491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11867.786088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11872.863362] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11877.350529] LustreError: 302686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11877.369861] LustreError: 302686:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11882.390491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11882.466870] LustreError: 302685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11882.781244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11887.577329] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11897.227706] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11904.104817] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11913.176208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11913.493849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11920.808851] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11927.847841] LustreError: 304515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11928.580812] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11933.154041] LustreError: 305145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11937.274811] LustreError: 304515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11937.304420] LustreError: 304515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11947.495642] LustreError: 304517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11947.519269] LustreError: 304517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11956.089173] loop1: detected capacity change from 0 to 204800000 [11956.262318] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11956.271616] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11956.285302] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11962.570852] loop1: detected capacity change from 0 to 204800000 [11965.195225] LustreError: 305145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11965.210792] LustreError: 305145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [11967.942810] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11968.107486] Lustre: fs11017-OST0000: new disk, initializing [11968.111842] Lustre: srv-fs11017-OST0000: No data found on store. Initialize space. [11968.117202] Lustre: 306446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11968.185379] Lustre: fs11017-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11974.690793] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [11984.190470] Lustre: server umount fs11017-OST0000 complete [11998.324730] loop1: detected capacity change from 0 to 204800000 [11998.699674] LustreError: 304515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11998.717356] LustreError: 304515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [12004.916371] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12005.259966] Lustre: fs11017-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12008.939956] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12011.931644] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [12014.068281] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12022.785989] loop2: detected capacity change from 0 to 204800000 [12023.160230] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12023.175585] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12023.204880] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12029.336622] loop2: detected capacity change from 0 to 204800000 [12035.445787] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12035.527075] LustreError: fs11017-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12035.543772] LustreError: fs11017-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12035.558367] LustreError: 309487:0:(tgt_mount.c:2734:server_fill_super()) Unable to start targets: -13 [12035.572758] LustreError: 309487:0:(tgt_mount.c:2192:server_put_super()) no obd fs11017-OST0001 [12035.585508] LustreError: 309487:0:(tgt_mount.c:130:server_deregister_mount()) fs11017-OST0001 not registered [12035.621637] Lustre: server umount fs11017-OST0001 complete [12035.626851] LustreError: 309487:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12047.547215] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12047.688996] Lustre: fs11017-OST0001: new disk, initializing [12047.691983] Lustre: srv-fs11017-OST0001: No data found on store. Initialize space. [12047.697373] Lustre: 310041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12047.820711] Lustre: fs11017-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12054.016376] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [12063.867120] Lustre: server umount fs11017-OST0000 complete [12065.249540] LustreError: 304515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12065.263181] LustreError: 304515:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [12074.175375] Lustre: server umount fs11017-OST0001 complete [12087.294873] loop1: detected capacity change from 0 to 204800000 [12094.176352] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12094.692930] Lustre: fs11017-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12101.694773] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [12110.491550] loop2: detected capacity change from 0 to 204800000 [12117.278916] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12126.380220] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [12137.829124] Lustre: server umount fs11017-OST0000 complete [12147.589548] Lustre: server umount fs11017-OST0001 complete [12162.646983] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12167.648510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12167.662129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12167.670233] LustreError: Skipped 1 previous similar message [12167.674873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12167.689034] Lustre: Skipped 1 previous similar message [12167.704220] Lustre: Skipped 1 previous similar message [12170.471712] Lustre: server umount lustre-OST0000 complete [12177.892059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12177.905747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12177.917262] Lustre: Skipped 1 previous similar message [12181.956830] Lustre: server umount lustre-MDT0000 complete [12186.740473] LustreError: 302666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788480642 with bad export cookie 14553200864005185732 [12186.749422] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12186.751107] LustreError: 302666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12187.383180] Lustre: server umount lustre-MDT0001 complete [12201.128840] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [12204.479625] Key type lgssc unregistered [12204.831620] LNet: 315966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12204.837604] LNetError: 315966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12204.873937] LNet: Removed LNI 192.168.201.123@tcp [12206.154841] Key type .llcrypt unregistered [12206.157265] Key type ._llcrypt unregistered [12240.748943] Key type ._llcrypt registered [12240.752812] Key type .llcrypt registered [12240.874383] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [12258.264808] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [12259.175943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12259.353032] alg: No test for adler32 (adler32-zlib) [12260.820093] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [12261.135255] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [12262.815511] Key type lgssc registered [12264.328599] Lustre: Echo OBD driver; http://www.lustre.org/ [12307.326521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12307.370985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12308.674113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12308.715856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12308.827885] Lustre: lustre-MDT0000: new disk, initializing [12308.882250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12308.895569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12313.299282] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [12324.721254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12324.785733] Lustre: 319825:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12324.843988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12324.847731] Lustre: Skipped 1 previous similar message [12324.900686] Lustre: lustre-MDT0001: new disk, initializing [12324.978142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12325.003633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12325.021122] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12330.168576] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [12340.984627] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12347.687586] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12355.226437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12355.499607] Lustre: lustre-OST0000: new disk, initializing [12355.505478] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12355.513813] Lustre: 321794:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12355.618717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12362.557600] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [12365.389518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12365.402630] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12365.525566] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12372.405853] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12374.310547] LustreError: 317996:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22 [12374.342835] Lustre: Mounted lustre-client - version 2.17.58_4_g0efcf2a [12375.512073] Lustre: Unmounted lustre-client [12389.868369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12389.881756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12389.888410] Lustre: Skipped 1 previous similar message [12392.621507] Lustre: server umount lustre-OST0000 complete [12396.513812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12396.521697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12396.530346] Lustre: Skipped 1 previous similar message [12396.539562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12400.097635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12400.099690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12400.124487] Lustre: Skipped 1 previous similar message [12402.422977] Lustre: server umount lustre-MDT0000 complete [12406.054068] LustreError: 319815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788480861 with bad export cookie 1653598349661335327 [12406.062190] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12406.071884] LustreError: 319815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12406.500697] Lustre: server umount lustre-MDT0001 complete [12419.772611] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [12422.930354] Key type lgssc unregistered [12430.303749] LNet: 323956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12430.323124] LNetError: 323956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12506.159103] LNet: Removed LNI 192.168.201.123@tcp [12506.892681] Key type .llcrypt unregistered [12506.895077] Key type ._llcrypt unregistered [12531.366531] Key type ._llcrypt registered [12531.370625] Key type .llcrypt registered [12531.491896] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [12548.129674] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [12549.218718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12549.268278] alg: No test for adler32 (adler32-zlib) [12550.414624] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [12550.605569] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [12552.255271] Key type lgssc registered [12553.529813] Lustre: Echo OBD driver; http://www.lustre.org/ [12587.916241] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 20:17:21 (1788481041) [12595.047329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12595.124222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12596.364585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12596.389761] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12596.479954] Lustre: lustre-MDT0000: new disk, initializing [12596.556092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12596.570243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12601.463378] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [12614.079335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12614.180810] Lustre: 327180:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12614.281594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12614.287772] Lustre: Skipped 1 previous similar message [12614.370723] Lustre: lustre-MDT0001: new disk, initializing [12614.517983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12614.568568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12614.581804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12620.152170] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [12630.589864] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12636.947769] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12644.676139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12645.020838] Lustre: lustre-OST0000: new disk, initializing [12645.024736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12645.032483] Lustre: 329149:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12645.168923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12651.813378] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [12654.701492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12654.714431] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12654.807096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12662.141697] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12680.676676] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12680.677211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12680.703323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12680.724388] Lustre: Skipped 1 previous similar message [12685.308371] Lustre: server umount lustre-OST0000 complete [12690.917583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12690.933603] Lustre: Skipped 1 previous similar message [12690.941164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12696.041291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12696.050425] Lustre: Skipped 2 previous similar messages [12696.735717] Lustre: server umount lustre-MDT0000 complete [12700.937567] LustreError: 327170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788481156 with bad export cookie 12715149164253940875 [12700.939699] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12700.955592] LustreError: 327170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12701.486359] Lustre: server umount lustre-MDT0001 complete [12714.127306] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [12716.713430] Key type lgssc unregistered [12716.999845] LNet: 331055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12717.016897] LNetError: 331055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12717.041255] LNet: Removed LNI 192.168.201.123@tcp [12717.671602] Key type .llcrypt unregistered [12717.673650] Key type ._llcrypt unregistered [12739.027271] Key type ._llcrypt registered [12739.028826] Key type .llcrypt registered [12739.119489] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [12740.319794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12740.444112] alg: No test for adler32 (adler32-zlib) [12741.499976] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [12741.722145] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [12743.399436] Key type lgssc registered [12744.344609] Lustre: Echo OBD driver; http://www.lustre.org/ [12751.277937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12751.296797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12752.585115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12756.794750] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [12763.107816] LustreError: 332324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12763.140380] LustreError: 332324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12764.816582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12765.200571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12769.361488] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [12776.997270] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12782.741453] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12789.624244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12789.734278] LustreError: 334145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12789.984828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12796.558082] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [12804.774150] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12815.014307] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12816.760672] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12821.473101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12821.490817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12821.511357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12823.025751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12823.028567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12823.038529] Lustre: Skipped 1 previous similar message [12826.877643] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.23@tcp (stopping) [12826.891179] Lustre: Skipped 2 previous similar messages [12832.004981] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.23@tcp (stopping) [12832.017706] Lustre: Skipped 3 previous similar messages [12835.295510] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12835.851567] Lustre: server umount lustre-MDT0000 complete [12837.114877] LustreError: 332319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12840.188633] LustreError: 332305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788481296 with bad export cookie 721847217336761494 [12840.201671] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12842.232738] LustreError: 334015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12842.241030] LustreError: 334015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12842.284263] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.23@tcp (stopping) [12842.286860] Lustre: Skipped 3 previous similar messages [12843.490979] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12843.503352] Lustre: Skipped 1 previous similar message [12846.493220] Lustre: server umount lustre-MDT0001 complete [12850.656772] Lustre: 335444:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [12851.076286] Lustre: server umount lustre-OST0000 complete [12860.435831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12864.999567] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [12868.127665] Lustre: 336510:0:(mgs_llog.c:1814:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.123@tcp [12870.585100] Lustre: server umount MGS complete [12879.864371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12880.518419] LustreError: 337080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12880.539499] LustreError: 337080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12880.605832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12886.712189] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [12891.114191] LustreError: 337081:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12891.138961] LustreError: 337081:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12898.674771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12899.083635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12904.981568] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [12915.573961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12915.917746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12923.896325] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [12934.634720] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12937.285886] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12944.617718] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12946.439614] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12955.105618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12955.114454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12955.150249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12955.157522] Lustre: Skipped 2 previous similar messages [12957.154981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12959.163413] Lustre: server umount lustre-OST0000 complete [12965.856879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12965.866283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12965.898308] Lustre: Skipped 1 previous similar message [12969.898399] Lustre: server umount lustre-MDT0000 complete [12972.522034] LustreError: 339479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12972.543407] LustreError: 339479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12974.375196] LustreError: 337063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788481430 with bad export cookie 721847217336762376 [12974.376137] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12974.398049] LustreError: 337063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [12975.391204] Lustre: server umount lustre-MDT0001 complete [12988.455605] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [12991.700239] Key type lgssc unregistered [12992.052875] LNet: 340520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12992.067391] LNetError: 340520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12992.096874] LNet: Removed LNI 192.168.201.123@tcp [12993.462655] Key type .llcrypt unregistered [12993.466037] Key type ._llcrypt unregistered [13015.935742] Key type ._llcrypt registered [13015.937507] Key type .llcrypt registered [13016.072687] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [13032.876657] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [13034.082776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13034.224234] alg: No test for adler32 (adler32-zlib) [13035.344535] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [13035.641864] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [13037.371292] Key type lgssc registered [13039.180724] Lustre: Echo OBD driver; http://www.lustre.org/ [13084.956210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13084.983232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13086.336962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13086.380872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13086.498193] Lustre: lustre-MDT0000: new disk, initializing [13086.552651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13086.572381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13091.541444] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [13101.923693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13102.082371] Lustre: 343679:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13102.210651] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13102.214971] Lustre: Skipped 1 previous similar message [13102.315125] Lustre: lustre-MDT0001: new disk, initializing [13102.452121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13102.485358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13102.508944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13107.398940] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [13118.448656] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13124.491592] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13132.512246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13132.843188] Lustre: lustre-OST0000: new disk, initializing [13132.852513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13132.860285] Lustre: 345649:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13132.946741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13139.561585] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [13141.085207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13141.094742] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13141.153236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13148.886391] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13154.819278] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13155.036948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13160.129405] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13160.367218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13167.078899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13167.080577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13167.111573] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13167.121821] Lustre: Skipped 1 previous similar message [13168.406503] Lustre: server umount lustre-OST0000 complete [13174.240947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13174.254378] LustreError: Skipped 1 previous similar message [13174.262940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13174.275019] Lustre: Skipped 1 previous similar message [13174.289060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13177.315339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13177.316472] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13178.934301] Lustre: server umount lustre-MDT0000 complete [13182.453552] LustreError: 343671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788481638 with bad export cookie 1081400024791854847 [13182.458705] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13182.473453] LustreError: 343671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13182.966907] Lustre: server umount lustre-MDT0001 complete [13192.448828] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 20:27:26 (1788481646) [13211.106540] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [13222.414797] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [13260.773453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13261.103324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13261.177545] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13261.314599] Lustre: lustre-MDT0000: new disk, initializing [13261.437375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13261.472460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13267.610446] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [13280.790554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13281.026959] Lustre: 350411:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13281.213550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13281.219906] Lustre: Skipped 1 previous similar message [13281.409565] Lustre: lustre-MDT0001: new disk, initializing [13281.535170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13281.572219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13281.592078] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13287.152406] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [13297.652920] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13303.030243] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13310.662429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13311.072353] Lustre: lustre-OST0000: new disk, initializing [13311.082990] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13311.100068] Lustre: 352379:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13311.208260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13316.562322] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [13318.722738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13318.729541] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13318.828327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13326.894599] Lustre: DEBUG MARKER: oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13365.221306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13365.236636] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13365.238372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13365.245780] LustreError: Skipped 1 previous similar message [13365.257537] Lustre: Skipped 2 previous similar messages [13369.608561] Lustre: server umount lustre-OST0000 complete [13373.925426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13373.930050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13373.949678] Lustre: Skipped 1 previous similar message [13373.963380] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13379.841935] Lustre: server umount lustre-MDT0000 complete [13380.576911] LustreError: 350423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13380.611109] LustreError: 350423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13384.383202] LustreError: 350404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788481840 with bad export cookie 1081400024791855743 [13384.391410] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13384.402335] LustreError: 350404:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [13384.801990] Lustre: server umount lustre-MDT0001 complete [13397.017130] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [13399.987090] Key type lgssc unregistered [13400.261764] LNet: 354554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13400.269325] LNetError: 354554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13400.282394] LNet: Removed LNI 192.168.201.123@tcp [13401.251972] Key type .llcrypt unregistered [13401.255415] Key type ._llcrypt unregistered [13497.441239] Key type ._llcrypt registered [13497.443654] Key type .llcrypt registered [13497.542736] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [13497.820538] Key type .llcrypt unregistered [13497.821782] Key type ._llcrypt unregistered [13579.737248] Key type ._llcrypt registered [13579.741110] Key type .llcrypt registered [13579.863939] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [13580.131335] Key type .llcrypt unregistered [13580.134247] Key type ._llcrypt unregistered [13612.604413] Key type ._llcrypt registered [13612.607603] Key type .llcrypt registered [13612.717768] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [13613.084686] Key type .llcrypt unregistered [13613.086657] Key type ._llcrypt unregistered [13646.336958] Key type ._llcrypt registered [13646.341786] Key type .llcrypt registered [13646.429785] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [13646.651800] Key type .llcrypt unregistered [13646.654246] Key type ._llcrypt unregistered [13726.962299] Key type ._llcrypt registered [13726.965741] Key type .llcrypt registered [13727.066309] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [13727.475367] Key type .llcrypt unregistered [13727.477596] Key type ._llcrypt unregistered [13767.023855] Key type ._llcrypt registered [13767.038167] Key type .llcrypt registered [13767.317962] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [13768.046651] Key type .llcrypt unregistered [13768.050825] Key type ._llcrypt unregistered [13809.523771] Key type ._llcrypt registered [13809.529993] Key type .llcrypt registered [13809.810667] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [13810.382936] Key type .llcrypt unregistered [13810.384488] Key type ._llcrypt unregistered [13847.103926] Key type ._llcrypt registered [13847.109847] Key type .llcrypt registered [13847.257763] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [13847.599364] Key type .llcrypt unregistered [13847.601824] Key type ._llcrypt unregistered [13884.920452] Key type ._llcrypt registered [13884.923218] Key type .llcrypt registered [13885.036256] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [13885.298523] Key type .llcrypt unregistered [13885.307511] Key type ._llcrypt unregistered [13921.079861] Key type ._llcrypt registered [13921.083475] Key type .llcrypt registered [13921.204861] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [13921.489507] Key type .llcrypt unregistered [13921.491832] Key type ._llcrypt unregistered [13988.206171] Key type ._llcrypt registered [13988.216991] Key type .llcrypt registered [13988.440561] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [13988.979309] Key type .llcrypt unregistered [13988.984407] Key type ._llcrypt unregistered [14022.523869] Key type ._llcrypt registered [14022.525928] Key type .llcrypt registered [14022.631322] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [14022.892246] Key type .llcrypt unregistered [14022.894140] Key type ._llcrypt unregistered [14060.092648] Key type ._llcrypt registered [14060.103686] Key type .llcrypt registered [14060.270388] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [14060.688384] Key type .llcrypt unregistered [14060.695682] Key type ._llcrypt unregistered [14124.358787] Key type ._llcrypt registered [14124.361245] Key type .llcrypt registered [14124.476957] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [14140.677664] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [14141.808294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14141.939721] alg: No test for adler32 (adler32-zlib) [14143.373383] Lustre: Lustre: Build Version: 2.17.58_4_g0efcf2a [14143.898306] LNet: Added LNI 192.168.201.123@tcp [8/256/0/180] [14145.633345] Key type lgssc registered [14147.537633] Lustre: Echo OBD driver; http://www.lustre.org/ [14193.992677] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14194.038461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14195.576229] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14195.614884] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14195.735341] Lustre: tst250fs-MDT0000: new disk, initializing [14195.840786] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14195.858545] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14202.336933] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [14217.057056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14217.185840] Lustre: 374858:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14217.334796] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14217.340728] Lustre: Skipped 1 previous similar message [14217.457473] Lustre: tst250fs-MDT0001: new disk, initializing [14217.614516] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14217.706187] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14217.718160] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14223.998107] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [14229.798857] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14239.705768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14240.144088] Lustre: tst250fs-OST0000: new disk, initializing [14240.152686] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14240.172446] Lustre: 376491:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14240.370260] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14246.692635] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14246.719907] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14246.892249] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14250.762897] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [14265.220335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14265.389134] Lustre: tst250fs-OST0001: new disk, initializing [14265.399599] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14265.407832] Lustre: 377362:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14265.521402] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14271.204735] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14271.226020] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14271.341579] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14274.999523] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_default_debug -1 all [14282.591737] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14285.463708] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14285.470582] Lustre: Skipped 1 previous similar message [14307.303311] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14307.315298] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14307.322856] Lustre: Skipped 1 previous similar message [14307.339210] Lustre: Skipped 1 previous similar message [14312.444383] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14312.447128] Lustre: Skipped 2 previous similar messages [14313.672615] Lustre: server umount tst250fs-MDT0000 complete [14318.381860] LustreError: 375652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788482774 with bad export cookie 752550841817494561 [14318.385603] LustreError: MGC192.168.201.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14318.393182] LustreError: 375652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14319.013721] Lustre: server umount tst250fs-MDT0001 complete [14328.228510] Lustre: 378901:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [14332.558939] Lustre: server umount tst250fs-OST0000 complete [14333.408635] Lustre: 378903:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [14338.528651] Lustre: 378901:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [14343.648448] Lustre: 378903:0:(ldlm_lib.c:1248:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [14346.973542] Lustre: server umount tst250fs-OST0001 complete [14366.161560] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [14375.281218] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [14412.625813] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 20:47:46 (1788482866) [14414.531348] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14434.039301] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing set_hostid [14448.576108] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing load_modules_local [14491.612172] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14167 sec ========= 20:49:05 (1788482945) [14493.723751] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:49:07 (1788482947) === [14497.602755] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:49:11 (1788482951) === [14543.760979] Lustre: DEBUG MARKER: oleg123-server.virtnet: executing unload_modules_local [14546.633267] Key type lgssc unregistered [14547.002060] LNet: 386501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14547.010671] LNetError: 386501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14547.026892] LNet: Removed LNI 192.168.201.123@tcp [14547.949336] Key type .llcrypt unregistered [14547.952560] Key type ._llcrypt unregistered