[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 490502108 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.988 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002397] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004020] kvm-guest: setup PV IPIs [ 0.006998] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 0.007027] Calibrating delay loop (skipped) preset value.. 4799.97 BogoMIPS (lpj=2399988) [ 0.008018] pid_max: default: 32768 minimum: 301 [ 0.009183] LSM: Security Framework initializing [ 0.011017] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013078] *** VALIDATE selinux *** [ 0.022776] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027719] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028170] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029130] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030142] *** VALIDATE tmpfs *** [ 0.031533] *** VALIDATE proc *** [ 0.033308] *** VALIDATE cgroup *** [ 0.034021] *** VALIDATE cgroup2 *** [ 0.035409] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040037] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.043673] debug: unmapping init [mem 0xffffffffa1a59000-0xffffffffa1a60fff] [ 0.045190] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046802] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047029] ... version: 2 [ 0.048016] ... bit width: 48 [ 0.049011] ... generic registers: 4 [ 0.050016] ... value mask: 0000ffffffffffff [ 0.051022] ... max period: 00007fffffffffff [ 0.052021] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.054332] rcu: Hierarchical SRCU implementation. [ 0.056697] smp: Bringing up secondary CPUs ... [ 0.057630] x86: Booting SMP configuration: [ 0.058034] .... node #0, CPUs: #1 #2 #3 [ 0.065353] smp: Brought up 1 node, 4 CPUs [ 0.067016] smpboot: Max logical packages: 1 [ 0.068011] smpboot: Total of 4 processors activated (19199.90 BogoMIPS) [ 0.267641] node 0 deferred pages initialised in 197ms [ 0.271180] devtmpfs: initialized [ 0.273322] x86/mm: Memory block size: 128MB [ 0.277109] gcov: version magic: 0x41383552 [ 0.280343] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.281076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.283402] pinctrl core: initialized pinctrl subsystem [ 0.285234] [ 0.285877] ************************************************************* [ 0.289016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.291024] ** ** [ 0.294017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.296014] ** ** [ 0.298015] ** This means that this kernel is built to expose internal ** [ 0.301014] ** IOMMU data structures, which may compromise security on ** [ 0.302012] ** your system. ** [ 0.305010] ** ** [ 0.306009] ** If you see this message and you are not debugging the ** [ 0.308011] ** kernel, report this immediately to your vendor! ** [ 0.310014] ** ** [ 0.313014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.315012] ************************************************************* [ 0.318626] NET: Registered protocol family 16 [ 0.321649] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.325101] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.327069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.331141] cpuidle: using governor menu [ 0.332816] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.336058] PCI: Using configuration type 1 for base access [ 0.338260] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.348067] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.349090] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.352047] cryptd: max_cpu_qlen set to 1000 [ 0.354437] ACPI: Added _OSI(Module Device) [ 0.356027] ACPI: Added _OSI(Processor Device) [ 0.358016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.359014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.364088] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.371584] ACPI: Interpreter enabled [ 0.373066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.374010] ACPI: Using IOAPIC for interrupt routing [ 0.375127] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.378422] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.391142] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.393064] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.395030] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.398123] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.405689] acpiphp: Slot [2] registered [ 0.407173] acpiphp: Slot [5] registered [ 0.408211] acpiphp: Slot [6] registered [ 0.410132] acpiphp: Slot [7] registered [ 0.411176] acpiphp: Slot [8] registered [ 0.413123] acpiphp: Slot [9] registered [ 0.415163] acpiphp: Slot [10] registered [ 0.417179] acpiphp: Slot [3] registered [ 0.418113] acpiphp: Slot [4] registered [ 0.420214] acpiphp: Slot [11] registered [ 0.422132] acpiphp: Slot [12] registered [ 0.424186] acpiphp: Slot [13] registered [ 0.426211] acpiphp: Slot [14] registered [ 0.428209] acpiphp: Slot [15] registered [ 0.430149] acpiphp: Slot [16] registered [ 0.431162] acpiphp: Slot [17] registered [ 0.433101] acpiphp: Slot [18] registered [ 0.434128] acpiphp: Slot [19] registered [ 0.436133] acpiphp: Slot [20] registered [ 0.437098] acpiphp: Slot [21] registered [ 0.439176] acpiphp: Slot [22] registered [ 0.440145] acpiphp: Slot [23] registered [ 0.442141] acpiphp: Slot [24] registered [ 0.443123] acpiphp: Slot [25] registered [ 0.445159] acpiphp: Slot [26] registered [ 0.446000] acpiphp: Slot [27] registered [ 0.446000] acpiphp: Slot [28] registered [ 0.449150] acpiphp: Slot [29] registered [ 0.450109] acpiphp: Slot [30] registered [ 0.451123] acpiphp: Slot [31] registered [ 0.453069] PCI host bridge to bus 0000:00 [ 0.454026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.456032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.459028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.461034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.464038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.467039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.469307] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.473317] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.477526] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.487957] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.493675] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.496038] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.498027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.503031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.506655] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.508838] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.512060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.515616] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.520000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.531021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.535032] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.540277] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.547026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.554028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.577023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.593667] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.601021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.611023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.643022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.657992] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.664016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.672028] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.695027] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.708596] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.717019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.730024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.746024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.756992] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.765024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.774019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.795020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.805135] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.815017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.824017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.856028] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.870793] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.873499] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.877560] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.880481] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.882307] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.886182] iommu: Default domain type: Passthrough [ 0.888450] SCSI subsystem initialized [ 0.890158] ACPI: bus type USB registered [ 0.892116] usbcore: registered new interface driver usbfs [ 0.894091] usbcore: registered new interface driver hub [ 0.896112] usbcore: registered new device driver usb [ 0.897146] pps_core: LinuxPPS API ver. 1 registered [ 0.899013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.902143] PTP clock support registered [ 0.904094] EDAC MC: Ver: 3.0.0 [ 0.907216] PCI: Using ACPI for IRQ routing [ 0.909193] NetLabel: Initializing [ 0.910012] NetLabel: domain hash size = 128 [ 0.911014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.912138] NetLabel: unlabeled traffic allowed by default [ 0.914194] vgaarb: loaded [ 0.916323] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.917020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.925485] clocksource: Switched to clocksource kvm-clock [ 1.036463] VFS: Disk quotas dquot_6.6.0 [ 1.038228] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.040395] *** VALIDATE ramfs *** [ 1.041875] *** VALIDATE hugetlbfs *** [ 1.043619] pnp: PnP ACPI init [ 1.046296] pnp: PnP ACPI: found 6 devices [ 1.061839] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.065380] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.067567] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.069718] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.072426] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.074462] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.077109] NET: Registered protocol family 2 [ 1.080140] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.085333] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.088994] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.094844] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.098732] TCP: Hash tables configured (established 65536 bind 65536) [ 1.102190] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.105879] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.109108] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.112346] NET: Registered protocol family 1 [ 1.115369] RPC: Registered named UNIX socket transport module. [ 1.117803] RPC: Registered udp transport module. [ 1.119902] RPC: Registered tcp transport module. [ 1.122068] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.125098] NET: Registered protocol family 44 [ 1.127229] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.129980] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.132781] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.135771] PCI: CLS 0 bytes, default 64 [ 1.138434] Unpacking initramfs... [ 2.570289] debug: unmapping init [mem 0xffff977d7cc54000-0xffff977d7ffbffff] [ 2.574576] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.576288] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.578978] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 3.075067] Initialise system trusted keyrings [ 3.078338] Key type blacklist registered [ 3.080954] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.091283] zbud: loaded [ 3.095333] *** VALIDATE nfs *** [ 3.096834] *** VALIDATE nfs4 *** [ 3.098477] pstore: using deflate compression [ 3.101946] Platform Keyring initialized [ 3.228835] NET: Registered protocol family 38 [ 3.230596] Key type asymmetric registered [ 3.231860] Asymmetric key parser 'x509' registered [ 3.233838] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.237370] io scheduler mq-deadline registered [ 3.239215] io scheduler kyber registered [ 3.241146] io scheduler bfq registered [ 3.243899] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.247549] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.250667] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.253579] ACPI: Power Button [PWRF] [ 3.260336] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.266713] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.278798] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.285170] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.297612] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.325639] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.354330] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.358867] Non-volatile memory driver v1.3 [ 3.361030] Linux agpgart interface v0.103 [ 3.401590] virtio_blk virtio1: [vda] 139480 512-byte logical blocks (71.4 MB/68.1 MiB) [ 3.405383] vda: detected capacity change from 0 to 71413760 [ 3.418937] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.421804] vdb: detected capacity change from 0 to 1073741824 [ 3.442765] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.447040] vdc: detected capacity change from 0 to 2621440000 [ 3.462729] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.466196] vdd: detected capacity change from 0 to 2621440000 [ 3.481867] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.484169] vde: detected capacity change from 0 to 4294967296 [ 3.496832] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.502445] vdf: detected capacity change from 0 to 4294967296 [ 3.507451] libphy: Fixed MDIO Bus: probed [ 3.515059] usbcore: registered new interface driver usbserial_generic [ 3.518376] usbserial: USB Serial support registered for generic [ 3.521437] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.526728] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.529083] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.532362] mousedev: PS/2 mouse device common for all mice [ 3.535290] rtc_cmos 00:05: RTC can wake from S4 [ 3.538638] rtc_cmos 00:05: registered as rtc0 [ 3.539808] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.541491] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.548622] intel_pstate: CPU model not supported [ 3.551927] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.556808] hid: raw HID events driver (C) Jiri Kosina [ 3.559370] usbcore: registered new interface driver usbhid [ 3.561959] usbhid: USB HID core driver [ 3.564109] drop_monitor: Initializing network drop monitor service [ 3.564163] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.567237] Initializing XFRM netlink socket [ 3.574989] NET: Registered protocol family 10 [ 3.578853] Segment Routing with IPv6 [ 3.580672] NET: Registered protocol family 17 [ 3.583499] mpls_gso: MPLS GSO support [ 3.589858] RAS: Correctable Errors collector initialized. [ 3.592196] AVX version of gcm_enc/dec engaged. [ 3.593728] AES CTR mode by8 optimization enabled [ 3.677966] sched_clock: Marking stable (3677939792, 0)->(4656928457, -978988665) [ 3.680949] registered taskstats version 1 [ 3.682521] Loading compiled-in X.509 certificates [ 3.684198] zswap: loaded using pool lzo/zbud [ 3.711227] Key type big_key registered [ 3.724489] Key type encrypted registered [ 3.726230] ima: No TPM chip found, activating TPM-bypass! [ 3.728151] ima: Allocated hash algorithm: sha1 [ 3.730037] ima: No architecture policies found [ 3.731982] evm: Initialising EVM extended attributes: [ 3.733952] evm: security.selinux [ 3.735275] evm: security.ima [ 3.736579] evm: security.capability [ 3.737924] evm: HMAC attrs: 0x1 [ 3.740602] rtc_cmos 00:05: setting system clock to 2026-06-12 18:04:59 UTC (1781287499) [ 3.747129] debug: unmapping init [mem 0xffffffffa2a03000-0xffffffffa2bfffff] [ 3.750455] debug: unmapping init [mem 0xffffffffa1782000-0xffffffffa1a58fff] [ 3.759826] Write protecting the kernel read-only data: 28672k [ 3.763545] debug: unmapping init [mem 0xffffffff9fe03000-0xffffffff9fffffff] [ 3.765950] debug: unmapping init [mem 0xffffffffa0714000-0xffffffffa07fffff] [ 3.802199] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.812616] systemd[1]: Detected virtualization kvm. [ 3.815158] systemd[1]: Detected architecture x86-64. [ 3.817330] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.842420] systemd[1]: No hostname configured. [ 3.843819] systemd[1]: Set hostname to . [ 3.845887] random: systemd: uninitialized urandom read (16 bytes read) [ 3.848646] systemd[1]: Initializing machine ID from random generator. [ 3.899147] random: ln: uninitialized urandom read (6 bytes read) [ 3.974553] random: systemd: uninitialized urandom read (16 bytes read) [ 3.977659] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.986191] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 3.990537] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Journal Service... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.657346] device-mapper: uevent: version 1.0.3 [ 4.659699] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.465710] random: fast init done [ 5.495714] virtio_net virtio0 ens2: renamed from eth0 [ 5.523268] scsi host0: ata_piix [ 5.552869] scsi host1: ata_piix [ 5.554652] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.557494] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.915448] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.242776] random: crng init done [ 10.244572] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.726057] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.959856] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.287488] SELinux: Disabled at runtime. [ 12.359596] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.369372] systemd[1]: Detected virtualization kvm. [ 12.371517] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.883391] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.886964] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.891802] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.899376] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.904796] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.912418] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.922778] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK [[ 13.054476] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.384346] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.730680] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.732580] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.892161] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.910652] EDAC sbridge: Ver: 1.1.2 [ 15.319701] Key type dns_resolver registered [ 15.619152] NFS: Registering the id_resolver key type [ 15.621170] Key type id_resolver registered [ 15.622461] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg458-server login: [ 67.948324] libcfs: loading out-of-tree module taints kernel. [ 68.001984] Key type ._llcrypt registered [ 68.004395] Key type .llcrypt registered [ 68.158576] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 87.867217] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 89.735416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 89.753681] alg: No test for adler32 (adler32-zlib) [ 91.246160] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 92.032872] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 93.848664] Key type lgssc registered [ 95.624790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 112.632798] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 155.756806] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 162.615131] hrtimer: interrupt took 3064500 ns [ 168.098041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 168.129353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.455417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 169.508974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 169.609317] Lustre: lustre-MDT0000: new disk, initializing [ 169.782134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 169.827542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 175.223440] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.895124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.082207] Lustre: 6506:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 190.179240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 190.190894] Lustre: Skipped 1 previous similar message [ 190.255256] Lustre: lustre-MDT0001: new disk, initializing [ 190.326493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 190.381127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 190.396174] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 196.010831] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.270076] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 212.284360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.750634] Lustre: lustre-OST0000: new disk, initializing [ 212.758513] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 212.766573] Lustre: 8446:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.900850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.042477] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.735581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 221.742996] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 221.838735] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 233.837503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.003591] Lustre: lustre-OST0001: new disk, initializing [ 234.009631] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 234.015595] Lustre: 9519:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.127782] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 241.537077] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.750623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 241.776415] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 241.863237] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 254.821531] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 260.201083] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 266.296960] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing check_logdir /tmp/testlogs/ [ 271.029758] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing yml_node [ 274.627561] Lustre: DEBUG MARKER: Client: 2.17.54.4 [ 277.015409] Lustre: DEBUG MARKER: MDS: 2.17.54.4 [ 279.542516] Lustre: DEBUG MARKER: OSS: 2.17.54.4 [ 281.206366] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jun 12 14:09:34 EDT 2026 [ 297.394400] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 298.655588] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 313.314667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 313.315741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 313.323582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 313.345962] Lustre: Skipped 3 previous similar messages [ 318.435098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 318.438182] Lustre: Skipped 6 previous similar messages [ 319.349323] Lustre: server umount lustre-MDT0000 complete [ 322.342767] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781287818 with bad export cookie 11935549732326037171 [ 322.355214] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 323.562112] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 323.580944] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 323.592605] Lustre: Skipped 1 previous similar message [ 323.617399] Lustre: Skipped 1 previous similar message [ 327.588463] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 327.592652] Lustre: Skipped 1 previous similar message [ 328.825201] Lustre: server umount lustre-MDT0001 complete [ 332.316492] Lustre: server umount lustre-OST0000 complete [ 336.303764] Lustre: server umount lustre-OST0001 complete [ 342.664462] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 352.034926] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 390.909267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 391.311722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 391.371667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 391.523618] Lustre: lustre-MDT0000: new disk, initializing [ 391.669189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 391.737097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 396.950727] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 410.671874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 410.923893] Lustre: 15074:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 411.074930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 411.080050] Lustre: Skipped 1 previous similar message [ 411.258588] Lustre: lustre-MDT0001: new disk, initializing [ 411.456872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 411.528544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 411.553384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 417.120200] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 427.877975] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 433.914745] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 442.721729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 443.156746] Lustre: lustre-OST0000: new disk, initializing [ 443.161891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 443.173676] Lustre: 17042:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 443.338664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 444.634265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 444.641566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 444.780373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 449.917479] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 459.207531] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 464.272250] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 464.629233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 470.109957] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 470.324284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 475.619535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.622682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 475.651920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 475.658469] Lustre: Skipped 1 previous similar message [ 478.728106] Lustre: server umount lustre-OST0000 complete [ 483.297602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 483.306675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 483.318084] Lustre: Skipped 1 previous similar message [ 485.862359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 485.868724] Lustre: Skipped 2 previous similar messages [ 489.341214] Lustre: server umount lustre-MDT0000 complete [ 490.978157] LustreError: 15084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.999964] LustreError: 15084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 493.356536] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781287989 with bad export cookie 11935549732326038704 [ 493.358573] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 493.369676] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 493.910588] Lustre: server umount lustre-MDT0001 complete [ 500.624476] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 14:13:13 (1781287993) [ 520.701778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.251756] LustreError: 19916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.326558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 526.311211] LustreError: 19917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.445159] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 531.426789] LustreError: 19916:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.360803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.621744] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 549.350900] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 554.985540] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 557.239483] Lustre: Failing over lustre-MDT0000 [ 557.487923] Lustre: server umount lustre-MDT0000 complete [ 560.358955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.360186] LustreError: 19911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.380929] Lustre: Skipped 2 previous similar messages [ 570.853416] LustreError: 19917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.896804] LustreError: 19917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 576.994199] Lustre: 3642:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781288056/real 1781288056] req@ffff977cc704f800 x1867815412983296/t0(0) o400->MGC192.168.204.158@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781288072 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 577.023491] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 578.945549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 586.273117] LustreError: 3640:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff977cc53ed180 x1867815412986240/t0(0) o250->MGC192.168.204.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 586.729641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 586.745699] Lustre: Skipped 1 previous similar message [ 591.842946] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 591.861755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 591.933912] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 592.290766] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 602.420799] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 611.298237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 611.307211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 611.320537] Lustre: Skipped 1 previous similar message [ 614.150438] Lustre: server umount lustre-MDT0000 complete [ 616.417376] LustreError: 22503:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 616.436170] LustreError: 22503:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 618.028915] LustreError: 19897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781288113 with bad export cookie 11935549732326040118 [ 618.040965] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 618.049837] LustreError: 19897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 618.515657] Lustre: server umount lustre-MDT0001 complete [ 627.258562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 632.322800] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 640.119427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 644.729765] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 652.675814] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 657.442373] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 659.711983] Lustre: Failing over lustre-MDT0000 [ 659.965707] Lustre: server umount lustre-MDT0000 complete [ 660.967130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 660.974411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 660.992312] LustreError: 23985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 661.003419] LustreError: 23985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 678.371142] Lustre: 3642:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781288158/real 1781288158] req@ffff977de7cb4e00 x1867815413013888/t0(0) o400->MGC192.168.204.158@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781288174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 678.406281] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 679.583087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 688.612386] LustreError: 3640:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff977dff4ca300 x1867815413016704/t0(0) o250->MGC192.168.204.158@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 689.085455] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 694.115706] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 694.246095] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 694.251130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 694.261748] Lustre: Skipped 1 previous similar message [ 694.275170] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 703.332778] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 708.578563] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 708.588362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 708.597549] Lustre: Skipped 2 previous similar messages [ 708.614556] Lustre: Skipped 2 previous similar messages [ 721.376255] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 721.558923] Lustre: server umount lustre-MDT0000 complete [ 725.500942] LustreError: 23238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781288221 with bad export cookie 11935549732326041056 [ 725.507232] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 725.511861] LustreError: 23238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 725.852040] Lustre: server umount lustre-MDT0001 complete [ 733.870260] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 14:17:07 (1781288227) [ 748.493536] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 756.152419] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 800.119330] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 811.823978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.095467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 812.135968] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 812.222970] Lustre: lustre-MDT0000: new disk, initializing [ 812.295466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 812.302671] Lustre: Skipped 3 previous similar messages [ 812.321266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 817.087787] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 827.517728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 827.623620] Lustre: 29632:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 827.918245] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 831.960238] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 840.076348] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 844.436263] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 850.527570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 850.780478] Lustre: 31600:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 852.250858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 852.276472] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 852.317172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 852.331951] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 856.322728] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 864.889450] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 870.167339] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 893.410077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 893.413234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 893.422391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 893.422404] Lustre: Skipped 6 previous similar messages [ 893.426975] LustreError: Skipped 1 previous similar message [ 893.468496] Lustre: Skipped 1 previous similar message [ 896.552833] Lustre: server umount lustre-OST0000 complete [ 906.609196] Lustre: server umount lustre-MDT0000 complete [ 908.771161] LustreError: 29645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 908.799545] LustreError: 29645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 910.301639] LustreError: 29627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781288406 with bad export cookie 11935549732326041462 [ 910.312251] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 910.325735] LustreError: 29627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 910.650748] Lustre: server umount lustre-MDT0001 complete [ 921.766507] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 925.448188] Key type lgssc unregistered [ 925.794928] LNet: 33582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.807362] LNetError: 33582:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.837195] LNet: Removed LNI 192.168.204.158@tcp [ 927.101177] Key type .llcrypt unregistered [ 927.105396] Key type ._llcrypt unregistered [ 955.029192] Key type ._llcrypt registered [ 955.033731] Key type .llcrypt registered [ 955.133675] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 955.978135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 956.077277] alg: No test for adler32 (adler32-zlib) [ 957.170022] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 957.413857] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 959.096200] Key type lgssc registered [ 960.130370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 968.300593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 968.344662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 969.888107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 974.834821] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 979.493838] LustreError: 34994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 979.535140] LustreError: 34994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 984.559471] LustreError: 34993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 985.947969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 986.374183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 991.294462] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 998.487836] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1003.998965] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1010.500121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1010.974050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1012.048100] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1015.105313] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1018.663909] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1028.241448] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1043.811746] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 14:22:16 (1781288536) [ 1046.130867] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1048.233257] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1055.369388] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 14:22:28 (1781288548) [ 1063.989499] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 14:22:37 (1781288557) [ 1065.264187] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1075.732402] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1075.740419] Lustre: Skipped 2 previous similar messages [ 1078.272288] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 14:22:50 (1781288570) [ 1080.497995] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1080.503618] Lustre: Skipped 2 previous similar messages [ 1089.005708] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 1089.016028] Lustre: Skipped 1 previous similar message [ 1107.199258] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 14:23:19 (1781288599) [ 1110.503060] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1114.488990] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1123.147273] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 14:23:36 (1781288616) [ 1125.062320] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1125.066248] Lustre: Skipped 9 previous similar messages [ 1138.145052] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 14:23:51 (1781288631) [ 1153.525647] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1153.528993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1153.536418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1153.543838] Lustre: Skipped 1 previous similar message [ 1153.557402] Lustre: Skipped 1 previous similar message [ 1158.638124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1158.653212] Lustre: Skipped 1 previous similar message [ 1159.541271] Lustre: server umount lustre-OST0000 complete [ 1163.747754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.750860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1163.759936] Lustre: Skipped 1 previous similar message [ 1163.773895] Lustre: Skipped 1 previous similar message [ 1168.868599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1168.877920] Lustre: Skipped 1 previous similar message [ 1169.997100] Lustre: server umount lustre-MDT0000 complete [ 1173.888459] LustreError: 34974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781288669 with bad export cookie 11826793864670289716 [ 1173.890317] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1173.896886] LustreError: 34974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1174.248042] Lustre: server umount lustre-MDT0001 complete [ 1185.743080] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 1188.763788] Key type lgssc unregistered [ 1189.086921] LNet: 42855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1189.097509] LNetError: 42855:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1189.116527] LNet: Removed LNI 192.168.204.158@tcp [ 1189.943430] Key type .llcrypt unregistered [ 1189.946153] Key type ._llcrypt unregistered [ 1208.542420] Key type ._llcrypt registered [ 1208.544930] Key type .llcrypt registered [ 1208.672766] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 1224.876779] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 1226.298883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1226.367805] alg: No test for adler32 (adler32-zlib) [ 1227.454351] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 1227.677508] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 1229.332732] Key type lgssc registered [ 1230.436787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1267.439577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1267.475858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.773352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1268.804608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1268.883452] Lustre: lustre-MDT0000: new disk, initializing [ 1269.044422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1269.078797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1273.726646] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1285.148622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1285.276543] Lustre: 46013:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1285.348906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1285.352761] Lustre: Skipped 1 previous similar message [ 1285.447495] Lustre: lustre-MDT0001: new disk, initializing [ 1285.570857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1285.637467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1285.653592] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1290.937711] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1301.232537] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1306.972955] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1313.789994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1314.014637] Lustre: lustre-OST0000: new disk, initializing [ 1314.018979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1314.033835] Lustre: 47980:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1314.121027] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1319.761551] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1323.618311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1323.646203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1323.795732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1329.896625] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1335.714488] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1335.908025] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1342.388675] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1342.779838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1349.601879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1349.611034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1349.652774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1349.667158] Lustre: Skipped 1 previous similar message [ 1351.676910] Lustre: server umount lustre-OST0000 complete [ 1359.841446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1359.845487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1359.863871] Lustre: Skipped 1 previous similar message [ 1359.888614] Lustre: Skipped 2 previous similar messages [ 1364.398183] Lustre: server umount lustre-MDT0000 complete [ 1368.834818] LustreError: 46002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781288864 with bad export cookie 15122621278921616654 [ 1368.838289] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1368.852398] LustreError: 46002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1369.383300] Lustre: server umount lustre-MDT0001 complete [ 1381.414545] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 14:27:54 (1781288874) [ 1390.270310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1391.002496] LustreError: 50441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.031535] LustreError: 50441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1391.170188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1395.949699] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1396.195402] LustreError: 50442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1400.297290] LustreError: 50441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.409388] LustreError: 50442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.744138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1406.212456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1411.462915] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1420.385155] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1426.214563] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1433.446828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1433.754411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1440.891698] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1449.784319] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1454.985100] LustreError: 52970:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1456.230572] Lustre: Modifying parameter general.timeout=1 in log params [ 1457.235794] Lustre: Modifying parameter general.timeout=2 in log params [ 1459.940606] Lustre: Modifying parameter general.timeout=5 in log params [ 1459.944227] Lustre: Skipped 2 previous similar messages [ 1464.684953] Lustre: Modifying parameter general.timeout=10 in log params [ 1464.693957] Lustre: Skipped 4 previous similar messages [ 1472.847675] Lustre: Modifying parameter general.timeout=18 in log params [ 1472.853733] Lustre: Skipped 7 previous similar messages [ 1489.015024] Lustre: Modifying parameter general.timeout=33 in log params [ 1489.017866] Lustre: Skipped 14 previous similar messages [ 1521.539228] Lustre: Modifying parameter general.timeout=61 in log params [ 1521.542562] Lustre: Skipped 27 previous similar messages [ 1586.672725] Lustre: Modifying parameter general.timeout=118 in log params [ 1586.690394] Lustre: Skipped 56 previous similar messages [ 1609.974500] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 14:31:43 (1781289103) [ 1613.452945] Lustre: 59931:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1615.504888] Lustre: 60035:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1617.845650] Lustre: 60133:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1622.301535] Lustre: 60347:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1622.313605] Lustre: 60347:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 1 previous similar message [ 1658.171696] Lustre: 61809:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1658.180252] Lustre: 61809:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 1 previous similar message [ 1658.214217] LustreError: 61809:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1662.908445] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 14:32:35 (1781289155) [ 1677.796300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1677.805922] LustreError: Skipped 1 previous similar message [ 1677.812249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1677.840580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1683.435279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1683.450138] Lustre: Skipped 5 previous similar messages [ 1683.767761] Lustre: server umount lustre-MDT0000 complete [ 1688.195584] LustreError: 50421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781289183 with bad export cookie 15122621278921617536 [ 1688.205072] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1688.549460] LustreError: 52391:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1688.554789] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.611679] Lustre: Skipped 2 previous similar messages [ 1688.762660] Lustre: server umount lustre-MDT0001 complete [ 1696.016113] Lustre: server umount lustre-OST0000 complete [ 1727.680103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1728.006425] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1728.039707] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1728.050591] Lustre: Skipped 34 previous similar messages [ 1728.459630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1733.232322] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1743.952748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.148677] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1744.232231] Lustre: 64341:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1749.873314] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1759.263850] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1766.883897] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1778.168818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1778.422323] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1779.055117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1779.065581] Lustre: Skipped 1 previous similar message [ 1787.415618] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1797.589619] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1799.642911] Lustre: 66910:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1810.420323] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1810.437264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1810.442779] LustreError: Skipped 1 previous similar message [ 1810.449927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1810.467668] Lustre: Skipped 1 previous similar message [ 1810.487458] Lustre: Skipped 1 previous similar message [ 1814.649719] Lustre: server umount lustre-OST0000 complete [ 1820.644215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1820.656627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1820.658871] Lustre: Skipped 1 previous similar message [ 1826.113144] Lustre: server umount lustre-MDT0000 complete [ 1830.744035] LustreError: 64333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781289326 with bad export cookie 15122621278921620770 [ 1830.772970] LustreError: 64333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1830.783914] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1831.185413] Lustre: server umount lustre-MDT0001 complete [ 1844.718345] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 1848.063365] Key type lgssc unregistered [ 1848.512734] LNet: 68121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1848.524801] LNetError: 68121:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1848.561833] LNet: Removed LNI 192.168.204.158@tcp [ 1849.830900] Key type .llcrypt unregistered [ 1849.833923] Key type ._llcrypt unregistered [ 1884.267343] Key type ._llcrypt registered [ 1884.269824] Key type .llcrypt registered [ 1884.354381] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 1885.501328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1885.562121] alg: No test for adler32 (adler32-zlib) [ 1886.887117] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 1887.323561] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 1888.984273] Key type lgssc registered [ 1890.765883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1902.142461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1902.188456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1903.971842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1909.738575] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1913.318162] LustreError: 69544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1913.356136] LustreError: 69544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1918.436985] LustreError: 69545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1921.335062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.073368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1928.275797] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1940.309193] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1949.165212] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1959.237108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1959.997344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1968.031751] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 1977.023649] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1987.114804] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1998.819148] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.839693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1998.845182] Lustre: Skipped 1 previous similar message [ 2003.427095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2003.434348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.481137] Lustre: Skipped 1 previous similar message [ 2003.497230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2004.741267] Lustre: server umount lustre-MDT0000 complete [ 2009.070953] LustreError: 70479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.092375] LustreError: 70479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2009.690250] LustreError: 69526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781289505 with bad export cookie 7893618873265134059 [ 2009.698849] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2009.704490] LustreError: 69526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2010.223465] Lustre: server umount lustre-MDT0001 complete [ 2024.893626] Lustre: server umount lustre-OST0000 complete [ 2058.968564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2059.470785] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2059.557555] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2059.878567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2065.086966] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2075.183913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.351774] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2075.419716] Lustre: 74418:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2075.916276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2081.795722] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2091.218515] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2098.728972] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2106.615871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2106.804778] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2107.277829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.381529] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2123.132200] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2124.889661] Lustre: 76995:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2126.964249] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2135.521807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2135.536220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2135.558353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2135.562975] Lustre: Skipped 2 previous similar messages [ 2139.649944] Lustre: server umount lustre-OST0000 complete [ 2146.276784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2146.277967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2146.305636] Lustre: Skipped 2 previous similar messages [ 2146.333627] Lustre: Skipped 3 previous similar messages [ 2150.386735] Lustre: server umount lustre-MDT0000 complete [ 2151.393058] LustreError: 77254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.423894] LustreError: 77254:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2154.384125] LustreError: 76287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781289650 with bad export cookie 7893618873265134906 [ 2154.392436] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2154.397836] LustreError: 76287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2154.704595] Lustre: server umount lustre-MDT0001 complete [ 2168.066871] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 2171.256780] Key type lgssc unregistered [ 2171.632599] LNet: 78210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2171.646109] LNetError: 78210:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2171.677941] LNet: Removed LNI 192.168.204.158@tcp [ 2172.638593] Key type .llcrypt unregistered [ 2172.646311] Key type ._llcrypt unregistered [ 2200.725286] Key type ._llcrypt registered [ 2200.728560] Key type .llcrypt registered [ 2200.881711] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2201.948241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2201.986116] alg: No test for adler32 (adler32-zlib) [ 2203.167706] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 2203.480522] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 2205.232194] Key type lgssc registered [ 2206.791844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2216.364220] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2216.379458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.968331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2223.186753] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2228.199682] LustreError: 79625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2228.248666] LustreError: 79625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2233.319160] LustreError: 79626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2234.914502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2235.477590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2240.817605] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2250.568742] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2257.051383] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2266.578710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2267.453363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2274.818074] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2284.594215] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2296.311480] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2297.551119] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2298.778888] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2301.407141] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2301.414400] Lustre: Skipped 1 previous similar message [ 2305.941878] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2305.950134] Lustre: Skipped 3 previous similar messages [ 2315.123339] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2315.127781] Lustre: Skipped 7 previous similar messages [ 2331.175632] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2331.181792] Lustre: Skipped 15 previous similar messages [ 2363.411336] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2363.420884] Lustre: Skipped 27 previous similar messages [ 2414.999982] Lustre: 87741:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2418.622700] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 14:45:11 (1781289911) [ 2420.465977] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2422.737828] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 14:45:15 (1781289915) [ 2428.389937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2428.396815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.408491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2431.461402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2431.465628] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2431.476393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2433.731586] Lustre: server umount lustre-OST0000 complete [ 2440.164044] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2440.180308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2440.197934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2440.205876] Lustre: Skipped 1 previous similar message [ 2444.570121] Lustre: server umount lustre-MDT0000 complete [ 2446.817723] LustreError: 79626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.860984] LustreError: 79626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2448.923321] LustreError: 83855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781289944 with bad export cookie 506914794858774975 [ 2448.933058] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.936831] LustreError: 83855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2449.303785] Lustre: server umount lustre-MDT0001 complete [ 2462.887258] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 2467.071366] Key type lgssc unregistered [ 2467.530820] LNet: 88993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2467.540516] LNetError: 88993:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2467.560151] LNet: Removed LNI 192.168.204.158@tcp [ 2468.593539] Key type .llcrypt unregistered [ 2468.600463] Key type ._llcrypt unregistered [ 2475.095698] Key type ._llcrypt registered [ 2475.099623] Key type .llcrypt registered [ 2475.218150] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2481.560732] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules [ 2482.517661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2482.570615] alg: No test for adler32 (adler32-zlib) [ 2483.814933] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 2484.240136] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 2486.016203] Key type lgssc registered [ 2487.310245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2488.517251] LustreError: 90205:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2508.962287] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2510.933685] LustreError: 90205:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2518.727747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2518.754433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2520.339095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.746072] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2530.787543] LustreError: 91223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.815487] LustreError: 91223:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2535.913786] LustreError: 91224:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.025975] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 14:47:08 (1781290028) [ 2541.029952] LustreError: 91223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.234212] LustreError: 91255:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2542.252764] LustreError: 91255:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2542.277552] LustreError: 91255:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 2542.699260] Lustre: server umount lustre-MDT0000 complete [ 2559.532060] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 2563.278908] Key type lgssc unregistered [ 2563.707635] LNet: 92751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2563.723572] LNetError: 92751:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2563.756893] LNet: Removed LNI 192.168.204.158@tcp [ 2564.746876] Key type .llcrypt unregistered [ 2564.759640] Key type ._llcrypt unregistered [ 2584.913809] Key type ._llcrypt registered [ 2584.919560] Key type .llcrypt registered [ 2585.114392] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2586.351474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2586.655618] alg: No test for adler32 (adler32-zlib) [ 2587.813996] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 2588.103891] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 2589.760782] Key type lgssc registered [ 2590.831474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2599.152947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2599.187742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2600.673425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.081442] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2611.180268] LustreError: 93970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2611.198512] LustreError: 93970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2613.633159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2614.407748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2619.244943] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2627.880789] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2635.076762] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2642.949689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2643.488732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.585308] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2661.957563] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2718.304992] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 14:50:11 (1781290211) [ 2720.174775] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2722.018297] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 14:50:15 (1781290215) [ 2726.890261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2726.900195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2726.911554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2730.467817] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2730.468522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2730.497035] Lustre: Skipped 1 previous similar message [ 2730.500841] Lustre: Skipped 1 previous similar message [ 2732.897121] Lustre: server umount lustre-MDT0000 complete [ 2735.586054] LustreError: 93970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2735.602593] LustreError: 93970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2736.768554] LustreError: 93952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781290232 with bad export cookie 13305686362084611705 [ 2736.786268] LustreError: 93952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2736.790764] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2737.232531] Lustre: server umount lustre-MDT0001 complete [ 2752.559804] Lustre: server umount lustre-OST0000 complete [ 2765.902577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2766.354206] LustreError: 98265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.608832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2771.718585] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2771.958814] LustreError: 98266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.063462] LustreError: 98265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2780.975196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2781.726633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2786.649501] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2794.264718] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2799.969875] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2813.577709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2813.884778] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2813.897482] LustreError: 98253:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2813.925211] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2813.933702] LustreError: 100348:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2813.941891] LustreError: 100348:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2813.951769] LustreError: 100348:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2814.051704] Lustre: server umount lustre-OST0000 complete [ 2814.059839] LustreError: 100348:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2824.452041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2824.701529] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2824.706822] LustreError: 98254:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2824.733319] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2824.745196] LustreError: 100815:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2824.752827] LustreError: 100815:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2824.758207] LustreError: 100815:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2824.844352] Lustre: server umount lustre-OST0000 complete [ 2824.848424] LustreError: 100815:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2836.071812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2836.326429] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2836.330185] Lustre: Found index 0 for lustre-OST0000, updating log [ 2836.335390] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2836.425839] Lustre: lustre-OST0000: new disk, initializing [ 2836.436209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2836.448596] Lustre: 101349:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2836.758613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2843.358909] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2847.347217] Lustre: Failing over lustre-OST0000 [ 2847.514379] Lustre: server umount lustre-OST0000 complete [ 2852.457301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2852.462400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2852.472438] Lustre: Skipped 1 previous similar message [ 2852.483499] Lustre: Skipped 2 previous similar messages [ 2857.954134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2857.962231] Lustre: Skipped 1 previous similar message [ 2862.113717] Lustre: server umount lustre-MDT0000 complete [ 2863.077486] LustreError: 98266:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2865.216126] LustreError: 100367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781290360 with bad export cookie 13305686362084614449 [ 2865.219067] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2865.231083] LustreError: 100367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2865.521849] Lustre: server umount lustre-MDT0001 complete [ 2873.636158] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 14:52:46 (1781290366) [ 2889.532662] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 2898.637254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2898.938867] LustreError: 103696:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.953089] LustreError: 103696:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2899.010670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2903.981540] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2912.765655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2917.348103] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2920.162772] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2926.466901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2927.155606] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2933.544800] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2937.377566] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2937.388890] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2937.403784] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2942.053487] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2942.135539] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2942.238948] Lustre: lustre-OST0001: new disk, initializing [ 2942.243110] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2942.249904] Lustre: 106014:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2942.436046] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:55 [ 2942.456571] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2942.482764] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2942.488798] Lustre: Skipped 2 previous similar messages [ 2943.540969] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2943.567603] Lustre: lustre-OST0000: Recovery over after 0:06, of 1 clients 1 recovered and 0 were evicted. [ 2943.567844] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2943.601865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2949.015767] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2950.250288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2950.260379] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2950.432740] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2956.945685] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2963.795509] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2969.584114] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2969.591244] LustreError: Skipped 1 previous similar message [ 2969.598770] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.613979] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2974.655861] Lustre: server umount lustre-MDT0001 complete [ 2976.230574] LustreError: 103693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2976.255572] LustreError: 103693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2986.994823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2987.193344] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2987.201814] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2987.302380] Lustre: 103684:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2987.620192] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.58@tcp (not set up) [ 2987.629230] Lustre: Skipped 4 previous similar messages [ 2989.312493] LustreError: 107559:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2989.324575] LustreError: 107559:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2989.333236] LustreError: 103688:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2989.423356] LustreError: 107569:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.204.158@tcp: cfg command failed: rc = -17 [ 2989.435160] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2989.435160] [ 2993.163743] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 2993.655476] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3002.817857] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 14:54:55 (1781290495) [ 3019.626990] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 3036.774781] Lustre: 109629:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3056.084140] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3060.016452] Lustre: 110764:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3146.739854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3146.749568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3146.764184] Lustre: Skipped 2 previous similar messages [ 3146.778238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3152.467767] Lustre: server umount lustre-MDT0000 complete [ 3155.802960] LustreError: 103678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781290651 with bad export cookie 13305686362084615443 [ 3155.803905] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3155.809516] LustreError: 103678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3156.027878] Lustre: server umount lustre-MDT0001 complete [ 3168.384469] Lustre: server umount lustre-OST0000 complete [ 3181.835065] Lustre: server umount lustre-OST0001 complete [ 3186.853945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3196.071140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3204.022493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3213.505815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3232.432258] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 3243.204700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3243.242059] Lustre: lustre-MDT0000: reset Object Index mappings [ 3243.758189] LustreError: 114718:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3243.770257] LustreError: 114718:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3243.889431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3243.900885] Lustre: Skipped 1 previous similar message [ 3249.038867] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3260.390753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3260.406218] Lustre: lustre-MDT0001: reset Object Index mappings [ 3267.171195] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3270.657560] Lustre: 115868:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3277.185773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3285.309160] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3285.679250] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3289.393357] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3294.668239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3300.843936] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3300.848494] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3301.215344] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3308.489873] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3312.190169] Lustre: 117742:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3362.029489] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 15:00:55 (1781290855) [ 3393.000812] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3393.005273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3393.007694] Lustre: Skipped 5 previous similar messages [ 3393.018233] Lustre: Skipped 11 previous similar messages [ 3395.945406] Lustre: server umount lustre-MDT0000 complete [ 3398.119581] LustreError: 115831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3398.153980] LustreError: 115831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 3399.679404] LustreError: 117744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781290895 with bad export cookie 13305686362084662525 [ 3399.686687] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3399.691353] LustreError: 117744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3406.218541] Lustre: server umount lustre-MDT0001 complete [ 3416.220126] Lustre: server umount lustre-OST0000 complete [ 3425.972198] Lustre: server umount lustre-OST0001 complete [ 3432.860176] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 3443.335189] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 3481.087035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3481.360741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3481.398398] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3481.470401] Lustre: lustre-MDT0000: new disk, initializing [ 3481.549546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3481.564638] Lustre: Skipped 3 previous similar messages [ 3481.580227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3485.454416] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3492.493759] Lustre: server umount lustre-MDT0000 complete [ 3501.599406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3501.852872] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3501.868357] Lustre: 122027:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3501.880881] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3506.490775] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3512.305914] Lustre: server umount lustre-MDT0000 complete [ 3521.498615] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 15:03:34 (1781291014) [ 3522.963699] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3524.553864] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 15:03:37 (1781291017) [ 3543.077755] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 3553.589664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3558.409698] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3566.977053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3567.154506] Lustre: 124076:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3567.209275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3567.213525] Lustre: Skipped 1 previous similar message [ 3567.283769] Lustre: lustre-MDT0001: new disk, initializing [ 3567.437160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3567.459958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3572.184203] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3577.111543] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3583.021751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3583.286176] Lustre: lustre-OST0000: new disk, initializing [ 3583.290328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3583.294347] Lustre: 125658:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3584.889780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3584.902971] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3585.070786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3589.891744] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3600.481981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3600.670558] Lustre: lustre-OST0001: new disk, initializing [ 3600.693914] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3600.698809] Lustre: 126527:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3602.133035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3602.141367] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3602.253547] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3607.569362] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 3617.249258] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3620.656969] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4041.888359] Lustre: 132946:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4041.894919] Lustre: 132946:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4041.903560] Lustre: 132946:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4041.911599] Lustre: 132946:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4041.917042] Lustre: 132946:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4041.926110] Lustre: 132946:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4352.373610] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 15:17:24 (1781291844) [ 4368.360333] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.368461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4368.376022] Lustre: Skipped 6 previous similar messages [ 4368.385594] Lustre: Skipped 7 previous similar messages [ 4374.690767] Lustre: server umount lustre-MDT0000 complete [ 4378.606245] LustreError: 124882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.625294] LustreError: 124882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4379.108944] LustreError: 132992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781291874 with bad export cookie 13305686362084731398 [ 4379.112082] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4379.131294] LustreError: 132992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4383.716203] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.736952] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4383.742386] Lustre: Skipped 1 previous similar message [ 4383.760715] Lustre: Skipped 5 previous similar messages [ 4385.723978] Lustre: server umount lustre-MDT0001 complete [ 4395.919932] Lustre: server umount lustre-OST0000 complete [ 4407.108734] Lustre: server umount lustre-OST0001 complete [ 4416.712386] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 4428.686809] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 4472.243610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4472.604842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4472.647871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4472.781292] Lustre: lustre-MDT0000: new disk, initializing [ 4472.939608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4472.947130] Lustre: Skipped 5 previous similar messages [ 4472.977544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4478.145568] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4490.733251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4490.892602] Lustre: 136004:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4490.967132] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4490.971104] Lustre: Skipped 1 previous similar message [ 4491.035555] Lustre: lustre-MDT0001: new disk, initializing [ 4491.235070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4491.258658] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4496.733650] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4508.850483] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4516.906172] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4525.864513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4526.186533] Lustre: lustre-OST0000: new disk, initializing [ 4526.191763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4526.197371] Lustre: 137974:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4526.284465] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4526.289687] Lustre: Skipped 1 previous similar message [ 4528.080766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4528.099244] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4528.215515] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4535.395165] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 4548.267870] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4559.972293] Lustre: lustre-MDD0000: changelog on [ 4563.981280] Lustre: lustre-MDD0001: changelog on [ 6887.379708] Lustre: lustre-MDD0001: changelog off [ 6889.435675] Lustre: 141201:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6890.328925] Lustre: lustre-MDD0000: changelog off [ 6898.658256] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6898.670164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6898.679414] LustreError: Skipped 1 previous similar message [ 6898.680676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6898.705943] Lustre: Skipped 1 previous similar message [ 6898.720152] Lustre: Skipped 3 previous similar messages [ 6903.057374] Lustre: server umount lustre-OST0000 complete [ 6907.880937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6907.884627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6907.902643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6908.905066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6914.019910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6914.064350] Lustre: Skipped 3 previous similar messages [ 6921.185497] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6921.550251] Lustre: server umount lustre-MDT0000 complete [ 6924.273399] LustreError: 136010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6924.302724] LustreError: 136010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6925.934037] LustreError: 135995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781294421 with bad export cookie 13305686362085507278 [ 6925.945558] LustreError: 135995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6925.948188] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6926.376886] Lustre: server umount lustre-MDT0001 complete [ 6939.160174] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 6942.169229] Key type lgssc unregistered [ 6942.482599] LNet: 147176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6942.489318] LNetError: 147176:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6942.507163] LNet: Removed LNI 192.168.204.158@tcp [ 6943.279307] Key type .llcrypt unregistered [ 6943.283659] Key type ._llcrypt unregistered [ 6965.307398] Key type ._llcrypt registered [ 6965.316360] Key type .llcrypt registered [ 6965.415571] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 6980.368111] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 6981.271191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6981.350988] alg: No test for adler32 (adler32-zlib) [ 6982.427369] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 6982.720785] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 6984.416167] Key type lgssc registered [ 6985.483356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7016.170896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7016.264915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7017.540725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7017.592653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7017.657321] Lustre: lustre-MDT0000: new disk, initializing [ 7017.751025] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7017.770186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7022.906101] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7033.938702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7034.084482] Lustre: 150385:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7034.131583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7034.136388] Lustre: Skipped 1 previous similar message [ 7034.255281] Lustre: lustre-MDT0001: new disk, initializing [ 7034.519591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7034.591285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7034.611594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7041.018203] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7051.485537] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7057.301994] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7063.785245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7064.076766] Lustre: lustre-OST0000: new disk, initializing [ 7064.084306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7064.098504] Lustre: 152354:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7064.241251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7070.253838] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7073.866367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7073.880957] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7074.008700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7079.632511] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7085.474972] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_module obdecho/obdecho [ 7087.515609] LustreError: 153356:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7087.526725] LustreError: 153356:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7089.271979] LustreError: 153454:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 7100.897885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7100.912711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7100.946744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7104.996604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7104.999337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7105.024660] Lustre: Skipped 1 previous similar message [ 7105.041469] Lustre: Skipped 2 previous similar messages [ 7106.993892] Lustre: server umount lustre-MDT0000 complete [ 7110.607877] LustreError: 151181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781294606 with bad export cookie 13830486007248716976 [ 7110.612139] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7110.614235] LustreError: 151181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7110.948064] Lustre: server umount lustre-MDT0001 complete [ 7125.067581] Lustre: server umount lustre-OST0000 complete [ 7148.782404] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 7158.233464] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 7197.280174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7197.715261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7197.770643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7197.931165] Lustre: lustre-MDT0000: new disk, initializing [ 7198.099196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7198.122749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7203.854909] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7214.763370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7214.870156] Lustre: 157174:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7214.954224] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7214.958122] Lustre: Skipped 1 previous similar message [ 7215.065222] Lustre: lustre-MDT0001: new disk, initializing [ 7215.190329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7215.232913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7215.245985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7220.117144] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7230.795502] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7236.580505] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7244.310044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7244.604762] Lustre: lustre-OST0000: new disk, initializing [ 7244.611496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7244.618201] Lustre: 159141:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7244.746075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7251.119723] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7254.665294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7254.675549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7254.798231] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7262.938758] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7269.221485] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7269.694936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7276.004274] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7276.377577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7280.611131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7280.622234] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7280.627648] LustreError: Skipped 1 previous similar message [ 7280.629478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7280.659481] Lustre: Skipped 1 previous similar message [ 7285.171098] Lustre: server umount lustre-OST0000 complete [ 7290.849736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7290.857108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7290.863701] Lustre: Skipped 1 previous similar message [ 7290.880289] Lustre: Skipped 1 previous similar message [ 7295.322584] Lustre: server umount lustre-MDT0000 complete [ 7295.970507] LustreError: 158746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7295.991876] LustreError: 158746:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7299.183144] LustreError: 157968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781294794 with bad export cookie 13830486007248717935 [ 7299.195164] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7299.201160] LustreError: 157968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7299.636356] Lustre: server umount lustre-MDT0001 complete [ 7303.413839] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 16:06:36 (1781294796) [ 7305.292782] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7307.065536] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 16:06:40 (1781294800) [ 7314.863125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7315.322246] LustreError: 161556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7315.395246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7320.333614] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7320.557263] LustreError: 161557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7325.670250] LustreError: 161556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7328.601091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7334.149432] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7344.852110] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7351.973400] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7359.295215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7359.750178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7359.756662] Lustre: Skipped 1 previous similar message [ 7366.111330] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7374.057231] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7384.547346] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7384.553717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7384.570056] Lustre: Skipped 1 previous similar message [ 7384.595746] Lustre: Skipped 2 previous similar messages [ 7386.279176] Lustre: server umount lustre-MDT0000 complete [ 7389.667209] LustreError: 161556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7389.712664] LustreError: 161556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7391.720547] LustreError: 164137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781294887 with bad export cookie 13830486007248718817 [ 7391.730640] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7391.733891] LustreError: 164137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7394.786403] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7394.803153] Lustre: Skipped 1 previous similar message [ 7394.808587] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7394.821959] Lustre: Skipped 1 previous similar message [ 7398.281865] Lustre: server umount lustre-MDT0001 complete [ 7405.193748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7419.431405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7419.898546] LustreError: 165400:0:(llog.c:1646:llog_backup()) MGC192.168.204.158@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7419.910386] Lustre: 165400:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.158@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7421.409766] LustreError: 165405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7421.429253] LustreError: 165405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7441.888987] LustreError: 165422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7441.906737] LustreError: 165422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 7477.227569] LustreError: 165406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7477.249851] LustreError: 165406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 7480.803874] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7480.818782] Lustre: 165400:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.158@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7521.794024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7526.904276] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7527.281231] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7536.632275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7537.136691] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xbfefc1bf1d122517 [ 7537.162460] Lustre: MGC192.168.204.158@tcp: Connection restored to 0@lo (at 0@lo) [ 7542.148556] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7542.758950] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7550.030264] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7551.882946] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7557.901221] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7559.687176] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7582.607193] Lustre: server umount lustre-MDT0000 complete [ 7583.718463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7583.721368] LustreError: 165406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7583.736413] Lustre: Skipped 2 previous similar messages [ 7583.808758] LustreError: 165406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 43 previous similar messages [ 7586.976983] LustreError: 164137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781295082 with bad export cookie 13830486007248725271 [ 7586.987189] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7586.993457] LustreError: 164137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7587.432866] Lustre: server umount lustre-MDT0001 complete [ 7602.125932] Lustre: server umount lustre-OST0000 complete [ 7626.758377] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 7638.357481] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 7688.450411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7688.838942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7688.872900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7688.942405] Lustre: lustre-MDT0000: new disk, initializing [ 7689.104967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7689.115187] Lustre: Skipped 1 previous similar message [ 7689.148698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7694.067963] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7707.927054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7708.084919] Lustre: 170507:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7708.414142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7714.176947] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7724.155167] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7730.135140] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7737.872543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7738.085785] Lustre: lustre-OST0000: new disk, initializing [ 7738.090839] Lustre: Skipped 1 previous similar message [ 7738.097396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7738.105270] Lustre: Skipped 2 previous similar messages [ 7738.112707] Lustre: 172478:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7744.197824] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7744.541861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7744.549174] Lustre: Skipped 1 previous similar message [ 7744.555399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7744.679440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7754.697280] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7760.841868] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7761.217583] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7767.424283] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7767.700560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7770.598649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7770.600657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7770.610743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7770.610756] Lustre: Skipped 2 previous similar messages [ 7770.624480] Lustre: Skipped 1 previous similar message [ 7770.632557] LustreError: Skipped 1 previous similar message [ 7776.321454] Lustre: server umount lustre-OST0000 complete [ 7785.445856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7787.667681] Lustre: server umount lustre-MDT0000 complete [ 7791.085624] LustreError: 170521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7791.112027] LustreError: 170521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7792.200497] LustreError: 170501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781295287 with bad export cookie 13830486007248742099 [ 7792.227585] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7792.852963] Lustre: server umount lustre-MDT0001 complete [ 7805.730292] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 16:14:57 (1781295297) [ 7813.990850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7820.243179] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7830.543442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7835.668896] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7844.921987] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7852.262321] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7859.603414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7867.951231] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 7877.671030] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7887.604828] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 16:16:20 (1781295380) [ 7897.059360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7897.065382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7897.069155] LustreError: Skipped 1 previous similar message [ 7897.088662] Lustre: Skipped 3 previous similar messages [ 7897.093733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7897.097534] Lustre: Skipped 5 previous similar messages [ 7898.393664] Lustre: server umount lustre-OST0000 complete [ 7908.763667] Lustre: server umount lustre-MDT0000 complete [ 7912.712681] LustreError: 175632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781295408 with bad export cookie 13830486007248742981 [ 7912.723847] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7912.729908] LustreError: 175632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7913.142298] Lustre: server umount lustre-MDT0001 complete [ 7925.094871] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 7928.145954] Key type lgssc unregistered [ 7928.475586] LNet: 178621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7928.482682] LNetError: 178621:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7928.502583] LNet: Removed LNI 192.168.204.158@tcp [ 7929.283650] Key type .llcrypt unregistered [ 7929.286642] Key type ._llcrypt unregistered [ 7948.081222] Key type ._llcrypt registered [ 7948.084027] Key type .llcrypt registered [ 7948.166613] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 7949.038791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7949.149038] alg: No test for adler32 (adler32-zlib) [ 7950.264521] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 7950.594328] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 7952.370751] Key type lgssc registered [ 7953.454721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7961.589527] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7961.615350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7977.184380] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7982.305910] LustreError: 179865:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.158@tcp: failed processing log, type 4: rc = -110 [ 7987.424430] LustreError: 179865:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7987.438788] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7987.454993] LustreError: MGC192.168.204.158@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7987.480114] LustreError: 179865:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7987.497886] LustreError: 179865:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 7987.509669] LustreError: 179865:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7987.624410] Lustre: server umount lustre-OST0000 complete [ 7987.631955] LustreError: 179865:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7997.973314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7999.522480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8005.024984] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8009.722168] LustreError: 180367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8009.752584] LustreError: 180367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8014.821019] LustreError: 180366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8014.898547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8015.305781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8019.875341] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8027.283629] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8033.740408] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8041.540625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8041.700580] LustreError: 182183:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8041.727218] Lustre: 182183:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.158@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8041.890328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8048.270571] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8056.361523] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8059.940152] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8059.944768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8059.968243] Lustre: Skipped 2 previous similar messages [ 8059.988651] Lustre: Skipped 2 previous similar messages [ 8064.998780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8065.012026] Lustre: Skipped 2 previous similar messages [ 8070.113528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8070.126061] Lustre: Skipped 2 previous similar messages [ 8073.184135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8073.517832] Lustre: server umount lustre-MDT0000 complete [ 8075.237786] LustreError: 180367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8075.262211] LustreError: 180367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8077.654391] LustreError: 182210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781295573 with bad export cookie 1073384575133048956 [ 8077.667464] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8078.432431] LustreError: 181096:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d1d826a7 x1867823653616256/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8078.644583] Lustre: server umount lustre-MDT0001 complete [ 8092.831403] Lustre: server umount lustre-OST0000 complete [ 8101.672769] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 8113.045807] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 8150.566593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8150.855961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8150.889699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8150.979703] Lustre: lustre-MDT0000: new disk, initializing [ 8151.075314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8151.094992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8155.890935] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8167.228258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8167.358473] Lustre: 185708:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8167.428141] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8167.434225] Lustre: Skipped 1 previous similar message [ 8167.586902] Lustre: lustre-MDT0001: new disk, initializing [ 8167.751405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8167.804394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8167.822549] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8172.364616] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8181.918598] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8187.637638] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8196.154493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8196.444914] Lustre: lustre-OST0000: new disk, initializing [ 8196.450449] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8196.459196] Lustre: 187675:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8196.565211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8203.626793] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8205.910906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8205.919200] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8206.029179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8214.283302] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8220.656953] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8220.944272] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8226.508984] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8226.960561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8231.912770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8231.913539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8231.928021] Lustre: Skipped 1 previous similar message [ 8231.934034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8231.954811] Lustre: Skipped 1 previous similar message [ 8235.741336] Lustre: server umount lustre-OST0000 complete [ 8239.596243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8239.602312] LustreError: Skipped 1 previous similar message [ 8239.605331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8239.619280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8242.147045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8251.363827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8251.373569] Lustre: Skipped 5 previous similar messages [ 8253.409865] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8253.619750] Lustre: server umount lustre-MDT0000 complete [ 8256.415027] LustreError: 185700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781295752 with bad export cookie 1073384575133049677 [ 8256.418772] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8256.422434] LustreError: 185700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8256.648267] Lustre: server umount lustre-MDT0001 complete [ 8265.649808] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 16:22:38 (1781295758) [ 8282.979952] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 8285.494657] Key type lgssc unregistered [ 8285.693812] LNet: 190623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8285.709127] LNetError: 190623:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8286.758463] LNet: Removed LNI 192.168.204.158@tcp [ 8287.597489] Key type .llcrypt unregistered [ 8287.599709] Key type ._llcrypt unregistered [ 8308.283996] Key type ._llcrypt registered [ 8308.287637] Key type .llcrypt registered [ 8308.415762] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 8309.529249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8309.570680] alg: No test for adler32 (adler32-zlib) [ 8310.662620] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 8310.999091] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 8312.682624] Key type lgssc registered [ 8314.023555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8322.686981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8322.727996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8324.349032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8329.581359] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8334.820442] LustreError: 191842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8334.842622] LustreError: 191842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8339.922066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8339.938573] LustreError: 191841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8340.293683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8346.057665] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8353.500745] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8360.817995] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8369.829219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8370.024587] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8370.029919] LustreError: 193709:0:(llog.c:1687:llog_backup()) MGC192.168.204.158@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8370.042989] Lustre: 193709:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.158@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8370.213784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8377.246757] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8385.462276] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8389.602042] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8389.615830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8389.621858] Lustre: Skipped 2 previous similar messages [ 8389.635201] Lustre: Skipped 2 previous similar messages [ 8394.731118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8399.848761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8399.864909] Lustre: Skipped 4 previous similar messages [ 8402.400174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8404.698625] Lustre: server umount lustre-MDT0000 complete [ 8404.963295] LustreError: 191836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8404.985613] LustreError: 191836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8410.082632] LustreError: 191842:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8410.116808] LustreError: 191842:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8410.340470] LustreError: 191823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781295906 with bad export cookie 14078312349129609647 [ 8410.348850] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8410.361217] LustreError: 191823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8410.969758] Lustre: server umount lustre-MDT0001 complete [ 8426.769918] Lustre: server umount lustre-OST0000 complete [ 8437.428450] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 8449.204539] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 8492.781489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8493.206326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8493.241988] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8493.350323] Lustre: lustre-MDT0000: new disk, initializing [ 8493.478295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8493.509280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8498.806310] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8509.678577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8509.826831] Lustre: 197236:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8509.901253] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8509.907273] Lustre: Skipped 1 previous similar message [ 8509.910201] LustreError: 198019:0:(llog.c:1687:llog_backup()) MGC192.168.204.158@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8509.916701] LustreError: 198019:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8509.920901] Lustre: 198019:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.158@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8509.954620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8509.971029] Lustre: Skipped 1 previous similar message [ 8510.077430] Lustre: lustre-MDT0001: new disk, initializing [ 8510.185807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8510.211884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8510.219364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8514.937973] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8524.380736] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8530.471528] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8538.002046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8538.227114] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8538.229162] Lustre: Skipped 1 previous similar message [ 8538.235934] LustreError: 199183:0:(llog.c:1687:llog_backup()) MGC192.168.204.158@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8538.241992] LustreError: 199183:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8538.246583] Lustre: 199183:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.158@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8538.253106] Lustre: 199183:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8538.271147] Lustre: lustre-OST0000: new disk, initializing [ 8538.275246] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8538.280964] Lustre: 199207:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8538.359730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8544.383927] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8544.824148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8544.832644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8544.969166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8554.402909] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8559.782544] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8559.989300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8564.496678] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8564.702055] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8570.855843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8570.856140] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8570.877620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8570.894519] LustreError: Skipped 1 previous similar message [ 8572.807109] Lustre: server umount lustre-OST0000 complete [ 8577.003602] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8577.019760] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8577.040268] Lustre: Skipped 1 previous similar message [ 8577.051284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8577.059978] Lustre: Skipped 1 previous similar message [ 8581.091052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8586.212718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8586.221159] Lustre: Skipped 3 previous similar messages [ 8591.328274] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8591.565663] Lustre: server umount lustre-MDT0000 complete [ 8595.426881] LustreError: 198369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8595.449389] LustreError: 198369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8595.767870] LustreError: 200748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781296091 with bad export cookie 14078312349129610368 [ 8595.783521] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8595.784312] LustreError: 200748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8596.150768] Lustre: server umount lustre-MDT0001 complete [ 8606.857116] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 16:28:19 (1781296099) [ 8625.533177] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 8637.244749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8637.787284] LustreError: 202339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8637.870227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8643.439577] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8651.793158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8657.167720] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8660.556449] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8668.142399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8675.564186] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8684.391385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8684.600986] Lustre: lustre-OST0001: new disk, initializing [ 8684.612771] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8684.628724] Lustre: 204648:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8684.770025] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8684.777266] Lustre: Skipped 2 previous similar messages [ 8691.266358] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8693.340925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8693.348061] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8693.441054] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8702.042356] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8709.171768] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8711.603335] Lustre: DEBUG MARKER: ADD OST3 [ 8717.386579] loop: module loaded [ 8717.514977] loop0: detected capacity change from 0 to 204800000 [ 8717.796978] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8717.825436] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8717.854796] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8725.170724] loop0: detected capacity change from 0 to 204800000 [ 8725.724964] Lustre: DEBUG MARKER: STOP OST3 [ 8732.674821] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8732.888751] Lustre: lustre-OST0003: new disk, initializing [ 8732.897301] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8732.904488] Lustre: 206693:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8734.803461] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8734.928863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8734.949010] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8735.010826] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8736.346344] Lustre: Failing over lustre-OST0003 [ 8736.480330] Lustre: server umount lustre-OST0003 complete [ 8736.864095] LustreError: 206835:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8739.984644] LustreError: 203824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8740.013515] LustreError: 203824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8740.344704] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8740.361846] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8740.777458] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8748.958481] Lustre: DEBUG MARKER: START OST3 again [ 8756.507799] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8756.861902] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8756.868346] Lustre: Skipped 1 previous similar message [ 8756.878321] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8758.568687] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8758.706774] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8758.707261] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8763.198646] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8784.708757] Lustre: Failing over lustre-OST0003 [ 8784.932861] Lustre: server umount lustre-OST0003 complete [ 8787.960104] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8787.970448] LustreError: 203822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8787.984291] Lustre: Skipped 2 previous similar messages [ 8788.012921] LustreError: 203822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 8805.860436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8805.873954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8805.891570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8805.899889] Lustre: Skipped 2 previous similar messages [ 8810.566091] Lustre: server umount lustre-MDT0000 complete [ 8815.082905] LustreError: 202319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781296310 with bad export cookie 14078312349129611250 [ 8815.086636] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8815.094443] LustreError: 202319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8821.605971] Lustre: server umount lustre-MDT0001 complete [ 8826.402473] Lustre: server umount lustre-OST0000 complete [ 8830.520263] Lustre: server umount lustre-OST0001 complete [ 8838.526738] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 8850.955752] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 8891.846623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8892.141631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8892.203416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8892.311466] Lustre: lustre-MDT0000: new disk, initializing [ 8892.390089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8892.419838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8896.883761] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8909.183678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8909.335765] Lustre: 212111:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8909.428567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8909.432418] Lustre: Skipped 1 previous similar message [ 8909.520895] Lustre: lustre-MDT0001: new disk, initializing [ 8909.712061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8909.733647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8914.643275] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8925.280574] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8931.846912] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8939.857724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8940.229268] Lustre: 214082:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8941.919945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8941.931428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8942.106939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8947.564334] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 8958.447935] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8964.080582] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8964.386597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8969.718488] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8969.965601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8972.780163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8972.795365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8972.806579] Lustre: Skipped 5 previous similar messages [ 8972.810330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8972.851701] Lustre: Skipped 9 previous similar messages [ 8978.145632] Lustre: server umount lustre-OST0000 complete [ 8988.786913] Lustre: server umount lustre-MDT0000 complete [ 8992.570427] LustreError: 212105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781296488 with bad export cookie 14078312349129613546 [ 8992.575086] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8992.582526] LustreError: 212105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8993.001569] Lustre: server umount lustre-MDT0001 complete [ 8997.207205] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 16:34:50 (1781296490) [ 9014.327150] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 9027.377827] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 9064.628950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9064.858576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9064.893933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9064.899908] Lustre: Skipped 1 previous similar message [ 9064.975695] Lustre: lustre-MDT0000: new disk, initializing [ 9064.978444] Lustre: Skipped 1 previous similar message [ 9065.051438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9069.520592] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9081.280320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9081.419660] Lustre: 218692:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9081.757307] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9086.564489] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9096.047833] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9101.120376] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9108.619087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9109.071743] Lustre: 220658:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9110.399084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9115.267171] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9125.747480] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9131.100692] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9131.409328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9136.953266] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9137.168662] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9141.227253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9141.236205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9141.246758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9141.246767] Lustre: Skipped 6 previous similar messages [ 9141.248455] LustreError: Skipped 2 previous similar messages [ 9141.290195] Lustre: Skipped 4 previous similar messages [ 9146.023449] Lustre: server umount lustre-OST0000 complete [ 9165.280131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9165.540517] Lustre: server umount lustre-MDT0000 complete [ 9166.817854] LustreError: 219500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9166.837662] LustreError: 219500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 17 previous similar messages [ 9169.723661] LustreError: 218684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781296665 with bad export cookie 14078312349129614428 [ 9169.724609] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9169.739968] LustreError: 218684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9170.073889] Lustre: server umount lustre-MDT0001 complete [ 9178.682309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9179.259596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9179.269744] Lustre: Skipped 5 previous similar messages [ 9184.111320] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9192.508341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9197.472102] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9204.703798] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9210.228263] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9217.583789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9225.039903] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9232.872921] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9244.647260] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9257.952158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9258.088563] Lustre: server umount lustre-OST0000 complete [ 9268.547085] Lustre: server umount lustre-MDT0000 complete [ 9272.352923] LustreError: 222855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781296768 with bad export cookie 14078312349129615310 [ 9272.366683] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9272.369869] LustreError: 222855:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9272.723311] Lustre: server umount lustre-MDT0001 complete [ 9372.344851] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [ 9375.534647] Key type lgssc unregistered [ 9376.061579] LNet: 226543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9376.073910] LNetError: 226543:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9376.111249] LNet: Removed LNI 192.168.204.158@tcp [ 9377.386308] Key type .llcrypt unregistered [ 9377.387959] Key type ._llcrypt unregistered [ 9411.081871] Key type ._llcrypt registered [ 9411.084281] Key type .llcrypt registered [ 9411.284779] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 9428.650805] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 9429.868444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9430.049152] alg: No test for adler32 (adler32-zlib) [ 9431.137114] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 9431.420792] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [ 9433.138354] Key type lgssc registered [ 9434.446509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9477.468149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9477.531229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9478.888512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9478.972087] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9479.113371] Lustre: lustre-MDT0000: new disk, initializing [ 9479.203629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9479.222828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9484.143587] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9495.837751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9495.962476] Lustre: 229916:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9496.024471] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9496.031659] Lustre: Skipped 1 previous similar message [ 9496.157803] Lustre: lustre-MDT0001: new disk, initializing [ 9496.301845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9496.354127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9496.371650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9502.301279] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9515.541473] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9522.948478] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9531.649349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9531.985369] Lustre: lustre-OST0000: new disk, initializing [ 9531.992443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9532.000212] Lustre: 231886:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9532.115956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9537.637599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9537.650408] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9537.864084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9540.647317] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9552.580603] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9557.190158] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9557.441377] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9563.207360] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9563.555656] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9568.742743] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9568.744141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9568.754271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9568.764466] Lustre: Skipped 1 previous similar message [ 9572.180437] Lustre: server umount lustre-OST0000 complete [ 9578.473401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9578.481608] LustreError: Skipped 1 previous similar message [ 9578.486882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9578.503792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9578.512787] Lustre: Skipped 1 previous similar message [ 9583.302927] Lustre: server umount lustre-MDT0000 complete [ 9587.398783] LustreError: 229907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781297083 with bad export cookie 6086974808104016231 [ 9587.400474] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9587.410313] LustreError: 229907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9587.924142] Lustre: server umount lustre-MDT0001 complete [ 9609.422888] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 9620.744091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9621.146430] LustreError: 234822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9621.166256] LustreError: 234822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9621.203263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9626.394317] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9626.599859] LustreError: 234823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9630.698649] LustreError: 234822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9635.809661] LustreError: 234823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9636.505432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9636.973150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9642.031335] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9645.409126] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9654.403912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9654.957481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9663.451143] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9676.211708] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9676.517858] Lustre: lustre-OST0001: new disk, initializing [ 9676.520640] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9676.546522] Lustre: 237131:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9676.751922] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9685.393987] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9686.698722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9686.719117] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9686.881911] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9698.572953] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9711.756310] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9728.374662] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 16:47:00 (1781297220) [ 9733.094495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9733.100731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9733.121622] Lustre: Skipped 3 previous similar messages [ 9733.131661] Lustre: Skipped 4 previous similar messages [ 9737.728751] Lustre: server umount lustre-MDT0000 complete [ 9738.227855] LustreError: 236941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9738.272612] LustreError: 236941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9742.124851] LustreError: 237130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781297237 with bad export cookie 6086974808104017113 [ 9742.137838] LustreError: 237130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9742.140217] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9742.537885] Lustre: server umount lustre-MDT0001 complete [ 9757.262797] Lustre: server umount lustre-OST0000 complete [ 9760.224180] Lustre: 228137:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781297239/real 1781297239] req@ffff977dfd1f7100 x1867825206647680/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781297255 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9760.283910] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9760.305694] Lustre: Skipped 1 previous similar message [ 9762.115108] Lustre: server umount lustre-OST0001 complete [ 9769.947384] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [ 9784.459769] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [ 9831.291709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9831.629822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9831.674797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9831.768396] Lustre: lustre-MDT0000: new disk, initializing [ 9831.866748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9831.884396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9836.886710] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9849.053256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9849.256467] Lustre: 241092:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9849.334550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9849.339462] Lustre: Skipped 1 previous similar message [ 9849.419664] Lustre: lustre-MDT0001: new disk, initializing [ 9849.654136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9849.671637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9854.560941] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9867.273988] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9873.056671] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9881.092035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9881.517306] Lustre: lustre-OST0000: new disk, initializing [ 9881.523646] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9881.530883] Lustre: 243066:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9882.819095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9882.848230] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9882.989507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9890.122715] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9905.051805] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9911.599925] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9911.817922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9917.357296] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9917.708556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9924.070896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9924.078656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9924.083111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9924.098354] Lustre: Skipped 1 previous similar message [ 9924.137317] Lustre: Skipped 1 previous similar message [ 9926.387506] Lustre: server umount lustre-OST0000 complete [ 9931.750116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9931.765520] LustreError: Skipped 1 previous similar message [ 9931.777121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9934.310265] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9934.340896] Lustre: Skipped 1 previous similar message [ 9937.405765] Lustre: server umount lustre-MDT0000 complete [ 9939.425800] LustreError: 241101:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9939.451707] LustreError: 241101:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9941.931594] LustreError: 241087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781297437 with bad export cookie 6086974808104019248 [ 9941.933412] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9941.943460] LustreError: 241087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9942.404392] Lustre: server umount lustre-MDT0001 complete [ 9953.698303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9954.362473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9954.370145] Lustre: Skipped 2 previous similar messages [ 9959.394495] LustreError: 245283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9959.415964] LustreError: 245283:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9960.025718] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9969.968353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9975.695821] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [ 9984.566751] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9991.356862] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10000.635766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10008.068191] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10017.515313] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10163.170623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10163.173498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10163.194448] Lustre: Skipped 1 previous similar message [10163.197139] Lustre: Skipped 2 previous similar messages [10168.260484] Lustre: server umount lustre-OST0000 complete [10175.459405] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10180.390988] Lustre: server umount lustre-MDT0000 complete [10184.162329] LustreError: 246011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10184.193278] LustreError: 246011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10184.742063] LustreError: 245264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781297680 with bad export cookie 6086974808104020130 [10184.753615] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10184.763000] LustreError: 245264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10185.214220] Lustre: server umount lustre-MDT0001 complete [10285.220773] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [10288.479653] Key type lgssc unregistered [10288.786186] LNet: 248902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10288.801617] LNetError: 248902:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10288.828972] LNet: Removed LNI 192.168.204.158@tcp [10290.123260] Key type .llcrypt unregistered [10290.125647] Key type ._llcrypt unregistered [10319.416727] Key type ._llcrypt registered [10319.419992] Key type .llcrypt registered [10319.576067] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [10336.795407] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [10337.874214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10338.061780] alg: No test for adler32 (adler32-zlib) [10339.194348] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [10339.440414] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [10341.144319] Key type lgssc registered [10342.511355] Lustre: Echo OBD driver; http://www.lustre.org/ [10405.331195] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [10420.175934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10420.230505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10421.764064] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10421.841624] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10422.039176] Lustre: lustre-MDT0000: new disk, initializing [10422.248135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10422.273175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10427.790145] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10438.540665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10438.628942] Lustre: 252994:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10438.691939] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10438.696402] Lustre: Skipped 1 previous similar message [10438.774354] Lustre: lustre-MDT0001: new disk, initializing [10438.879474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10438.915479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10438.933593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10443.514809] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10449.048444] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10456.635218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10456.944132] Lustre: lustre-OST0000: new disk, initializing [10456.947939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10456.955906] Lustre: 254623:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10457.077661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10460.495232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10460.513526] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10460.699555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10464.278259] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10475.175361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10475.421468] Lustre: lustre-OST0001: new disk, initializing [10475.425146] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10475.433447] Lustre: 255496:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10475.535351] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10482.354691] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10484.374802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10484.386295] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10484.539362] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10495.062346] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10503.696680] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10520.074157] Lustre: server umount lustre-MDT0000 complete [10520.559325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10520.570772] Lustre: Skipped 3 previous similar messages [10524.921355] LustreError: 253788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781298020 with bad export cookie 13400256152258042850 [10524.928747] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10524.950414] LustreError: 253788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10525.355530] Lustre: server umount lustre-MDT0001 complete [10540.384081] Lustre: server umount lustre-OST0000 complete [10541.856367] Lustre: 250477:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781298021/real 1781298021] req@ffff977ccea32a00 x1867826158711168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781298037 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10541.894386] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10545.779201] Lustre: server umount lustre-OST0001 complete [10561.687986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10572.001561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10584.284284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10597.430577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10621.176400] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [10633.043465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10633.088562] Lustre: lustre-MDT0000: reset Object Index mappings [10633.755664] LustreError: 260523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10633.778852] LustreError: 260523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10633.886661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10638.784410] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10639.331766] LustreError: 260524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10643.441023] LustreError: 260523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10648.550479] LustreError: 260524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10648.905708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10648.950805] Lustre: lustre-MDT0001: reset Object Index mappings [10649.345183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10654.842319] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10658.485756] Lustre: 261665:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10665.009436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10665.495230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10671.756750] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10674.730760] LustreError: 262022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10678.856162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10682.123105] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10687.501336] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10688.847573] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10696.537639] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10700.711942] Lustre: 263536:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10707.943699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10707.949041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10707.977246] Lustre: Skipped 1 previous similar message [10707.988629] Lustre: Skipped 3 previous similar messages [10708.412746] Lustre: server umount lustre-MDT0000 complete [10712.498626] LustreError: 260503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781298208 with bad export cookie 13400256152258049633 [10712.511352] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10712.524452] LustreError: 260503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10713.026214] Lustre: server umount lustre-MDT0001 complete [10727.981789] Lustre: server umount lustre-OST0000 complete [10728.289077] Lustre: 250480:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781298208/real 1781298208] req@ffff977de9502680 x1867826158760576/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781298224 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10728.332098] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10728.367785] Lustre: Skipped 2 previous similar messages [10732.730631] Lustre: server umount lustre-OST0001 complete [10758.316647] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 17:04:11 (1781298251) [10779.587063] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [10795.650427] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [10839.313584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10839.557713] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10839.662094] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10839.861940] Lustre: lustre-MDT0000: new disk, initializing [10839.983807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10839.992674] Lustre: Skipped 1 previous similar message [10840.014675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10845.041484] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10857.114197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10857.267288] Lustre: 267814:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10857.329116] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10857.339792] Lustre: Skipped 1 previous similar message [10857.426367] Lustre: lustre-MDT0001: new disk, initializing [10857.569920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10857.580281] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10862.285821] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10873.028938] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10880.568457] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10888.852541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10889.243776] Lustre: lustre-OST0000: new disk, initializing [10889.246921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10889.266041] Lustre: 269783:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10890.860658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10890.875392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10891.174096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10897.940102] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [10908.287955] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10913.019658] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10913.175495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10918.431595] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10918.727666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10921.971599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10921.992848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10922.004860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10927.076750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10927.081990] Lustre: Skipped 2 previous similar messages [10932.200281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10932.213063] Lustre: Skipped 1 previous similar message [10935.266346] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10935.515749] Lustre: server umount lustre-OST0000 complete [10942.435033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10942.462406] Lustre: Skipped 1 previous similar message [10942.468730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10942.478795] Lustre: Skipped 1 previous similar message [10952.674720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10952.685371] Lustre: Skipped 3 previous similar messages [10954.208295] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10954.635246] Lustre: server umount lustre-MDT0000 complete [10957.797677] LustreError: 267824:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10957.822883] LustreError: 267824:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10958.893369] LustreError: 267806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781298454 with bad export cookie 13400256152258051733 [10958.894296] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10958.907257] LustreError: 267806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10959.304720] Lustre: server umount lustre-MDT0001 complete [10983.448808] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [10994.948948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10995.538041] LustreError: 272732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10995.625974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10995.642696] Lustre: Skipped 2 previous similar messages [11000.870277] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11011.473419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11017.842364] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11021.791426] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11029.198916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11036.448466] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11045.935103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11046.152726] Lustre: lustre-OST0001: new disk, initializing [11046.162608] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11046.169864] Lustre: 275042:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11047.580440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11047.593138] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11047.725724] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11053.285877] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11064.612874] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11068.930210] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11075.629331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11075.665174] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11083.768139] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11083.768378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11083.772423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11083.784721] Lustre: Skipped 3 previous similar messages [11089.762693] Lustre: server umount lustre-MDT0000 complete [11094.002168] LustreError: 273672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11094.014364] LustreError: 273672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [11094.281258] LustreError: 272714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781298590 with bad export cookie 13400256152258052608 [11094.296744] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11094.594335] Lustre: server umount lustre-MDT0001 complete [11108.575802] Lustre: server umount lustre-OST0000 complete [11123.088341] Lustre: server umount lustre-OST0001 complete [11143.143525] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [11154.981252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11155.449272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11155.453518] Lustre: Skipped 3 previous similar messages [11160.576524] LustreError: 278100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11160.591693] LustreError: 278100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11160.704447] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11171.866945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11177.412509] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11180.988408] Lustre: 279240:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11188.839879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11196.811691] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11200.705368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11207.766601] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11213.359059] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11215.728659] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11225.094129] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11229.082213] Lustre: 281119:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11235.300847] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11235.320961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11247.474885] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 17:12:20 (1781298740) [11264.485800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11264.497048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11264.515163] Lustre: Skipped 2 previous similar messages [11264.533501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11264.546541] Lustre: Skipped 10 previous similar messages [11269.776435] Lustre: server umount lustre-MDT0000 complete [11273.493381] LustreError: 278081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781298769 with bad export cookie 13400256152258059188 [11273.498255] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11273.509309] LustreError: 278081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11280.012097] Lustre: server umount lustre-MDT0001 complete [11298.272166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11298.350330] Lustre: server umount lustre-OST0000 complete [11302.864138] Lustre: server umount lustre-OST0001 complete [11314.719923] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [11329.689798] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [11387.386344] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [11399.568470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11399.997381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11400.032961] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11400.157967] Lustre: lustre-MDT0000: new disk, initializing [11400.379070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11405.619934] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11419.940089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11420.210277] Lustre: 285399:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11420.685167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11420.694643] Lustre: Skipped 4 previous similar messages [11420.780281] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11426.687480] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11432.759738] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11442.038266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11442.561191] Lustre: 287036:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11443.734436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11449.532121] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11463.115704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11463.336629] Lustre: 287906:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11464.686544] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11464.697851] Lustre: Skipped 1 previous similar message [11464.888866] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11471.424510] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11484.034686] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11510.585158] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 17:16:42 (1781299002) [11515.879862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11515.902727] Lustre: Skipped 7 previous similar messages [11515.903665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11515.923949] Lustre: Skipped 11 previous similar messages [11520.151725] Lustre: server umount lustre-MDT0000 complete [11520.996710] LustreError: 285410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11521.037878] LustreError: 285410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [11524.096652] LustreError: 286195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781299019 with bad export cookie 13400256152258066832 [11524.112466] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11524.113808] LustreError: 286195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11524.549144] Lustre: server umount lustre-MDT0001 complete [11539.449489] Lustre: server umount lustre-OST0000 complete [11541.472613] Lustre: 250480:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781299021/real 1781299021] req@ffff977dffc94000 x1867826159038080/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781299037 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11543.715305] Lustre: server umount lustre-OST0001 complete [11554.375572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11559.853984] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11569.764161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11575.222233] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11584.174697] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11590.607556] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11598.091918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11606.597971] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11616.987828] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11638.619734] loop1: detected capacity change from 0 to 204800000 [11638.887157] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11638.905867] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11638.919163] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11644.264488] loop1: detected capacity change from 0 to 204800000 [11649.444955] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11649.538142] LustreError: 290147:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs29383: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11649.552691] LustreError: 290147:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs29383 [11649.557726] LustreError: 290147:0:(mgs_handler.c:517:mgs_target_reg()) fs29383-MDT0001: Error checking target index: rc = -13 [11649.574533] LustreError: fs29383-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11649.580932] LustreError: 293875:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [11649.594792] LustreError: 293875:0:(tgt_mount.c:2084:server_put_super()) no obd fs29383-MDT0001 [11649.604461] LustreError: 293875:0:(tgt_mount.c:116:server_deregister_mount()) fs29383-MDT0001 not registered [11649.666941] Lustre: server umount fs29383-MDT0001 complete [11649.670187] LustreError: 293875:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11658.588313] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11658.723367] Lustre: Setting parameter fs29383-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs29383-MDT0001 [11658.732601] Lustre: Skipped 1 previous similar message [11658.774878] Lustre: srv-fs29383-MDT0001: No data found on store. Initialize space. [11658.780834] Lustre: Skipped 4 previous similar messages [11658.854772] Lustre: fs29383-MDT0001: new disk, initializing [11658.857956] Lustre: Skipped 3 previous similar messages [11658.972757] Lustre: srv-fs29383-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11658.982662] Lustre: 294342:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29383-MDT0001-mdtlov: can't create new llog: rc = -115 [11658.991676] LustreError: 294342:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29383-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11663.387462] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11670.587736] Lustre: server umount fs29383-MDT0001 complete [11681.778055] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11685.912833] loop1: detected capacity change from 0 to 204800000 [11692.646156] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11692.923455] Lustre: 295910:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29383-MDT0001-mdtlov: can't create new llog: rc = -115 [11692.937909] LustreError: 295910:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29383-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11697.131697] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11697.320924] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11702.122068] Lustre: server umount fs29383-MDT0001 complete [11712.483959] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11712.488663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11712.509221] Lustre: Skipped 2 previous similar messages [11712.527502] Lustre: Skipped 2 previous similar messages [11726.304682] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11726.946764] Lustre: server umount lustre-MDT0000 complete [11738.852376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11739.057588] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11744.728917] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11744.761084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11754.139122] loop1: detected capacity change from 0 to 204800000 [11760.143824] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11760.471377] Lustre: 298359:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29383-MDT0001-mdtlov: can't create new llog: rc = -115 [11760.480887] LustreError: 298359:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29383-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11765.391587] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11771.120824] Lustre: server umount fs29383-MDT0001 complete [11778.532718] LustreError: 291985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11778.553127] LustreError: 291985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 111 previous similar messages [11795.937453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11801.954402] Lustre: server umount lustre-OST0000 complete [11812.147626] Lustre: server umount lustre-MDT0000 complete [11816.078496] LustreError: 294887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781299311 with bad export cookie 13400256152258069513 [11816.096082] LustreError: 294887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11816.103726] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11816.565792] Lustre: server umount lustre-MDT0001 complete [11828.375215] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [11831.107893] Key type lgssc unregistered [11831.380535] LNet: 300703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11831.395243] LNetError: 300703:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11831.419529] LNet: Removed LNI 192.168.204.158@tcp [11832.375162] Key type .llcrypt unregistered [11832.377380] Key type ._llcrypt unregistered [11853.726194] Key type ._llcrypt registered [11853.728329] Key type .llcrypt registered [11853.847969] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [11854.879167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11854.994813] alg: No test for adler32 (adler32-zlib) [11856.128872] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [11856.344048] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [11858.048704] Key type lgssc registered [11859.436248] Lustre: Echo OBD driver; http://www.lustre.org/ [11867.155646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11867.173849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11868.706422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11873.548105] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11878.886904] LustreError: 301991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11878.916803] LustreError: 301991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11882.495305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11882.913301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11888.045521] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11896.322123] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11902.338071] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11909.152908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11909.572964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11914.680181] LustreError: 303820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11917.115615] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11919.846560] LustreError: 304073:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11924.972081] LustreError: 303823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11925.516782] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11932.738201] LustreError: 303823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11932.755493] LustreError: 303823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11942.984127] LustreError: 304073:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11943.014509] LustreError: 304073:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11947.968636] loop1: detected capacity change from 0 to 204800000 [11948.185463] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11948.196126] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11948.204928] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11953.774753] loop1: detected capacity change from 0 to 204800000 [11959.673659] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11959.820374] Lustre: fs16578-OST0000: new disk, initializing [11959.836857] Lustre: srv-fs16578-OST0000: No data found on store. Initialize space. [11959.843405] Lustre: 305727:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11959.930048] Lustre: fs16578-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11963.453839] LustreError: 303823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11963.484141] LustreError: 303823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [11966.937383] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [11976.043773] Lustre: server umount fs16578-OST0000 complete [11988.986261] loop1: detected capacity change from 0 to 204800000 [11994.601879] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11994.711733] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11994.943214] Lustre: fs16578-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11999.294975] LustreError: 303822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11999.344806] LustreError: 303822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 20 previous similar messages [12001.968822] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12009.979037] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12011.810758] loop2: detected capacity change from 0 to 204800000 [12012.076932] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12012.095948] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12012.113791] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12017.778211] loop2: detected capacity change from 0 to 204800000 [12023.789988] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12023.933048] LustreError: fs16578-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12023.960092] LustreError: fs16578-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12023.974411] LustreError: 308766:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [12023.993152] LustreError: 308766:0:(tgt_mount.c:2084:server_put_super()) no obd fs16578-OST0001 [12024.000730] LustreError: 308766:0:(tgt_mount.c:116:server_deregister_mount()) fs16578-OST0001 not registered [12024.045404] Lustre: server umount fs16578-OST0001 complete [12024.052374] LustreError: 308766:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [12036.485322] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12036.688071] Lustre: fs16578-OST0001: new disk, initializing [12036.697494] Lustre: srv-fs16578-OST0001: No data found on store. Initialize space. [12036.723181] Lustre: 309319:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12036.844785] Lustre: fs16578-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12043.224366] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12052.306536] Lustre: server umount fs16578-OST0000 complete [12063.210837] Lustre: server umount fs16578-OST0001 complete [12065.875383] LustreError: 303822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.58@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12065.901731] LustreError: 303822:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [12076.901769] loop1: detected capacity change from 0 to 204800000 [12083.840322] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12084.184845] Lustre: fs16578-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12091.357350] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12100.094588] loop2: detected capacity change from 0 to 204800000 [12106.989275] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12113.930784] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12123.652597] Lustre: server umount fs16578-OST0000 complete [12132.267040] Lustre: server umount fs16578-OST0001 complete [12145.125753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12145.137888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12145.171074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12148.199436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12148.200451] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12148.201993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12148.231618] LustreError: Skipped 1 previous similar message [12151.168921] Lustre: server umount lustre-OST0000 complete [12158.435913] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12158.460742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12158.469441] Lustre: Skipped 2 previous similar messages [12161.644559] Lustre: server umount lustre-MDT0000 complete [12166.097244] LustreError: 301971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781299661 with bad export cookie 3218028397979387191 [12166.100187] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12166.122344] LustreError: 301971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12166.636690] Lustre: server umount lustre-MDT0001 complete [12178.268458] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [12180.938491] Key type lgssc unregistered [12181.208875] LNet: 315246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12181.222206] LNetError: 315246:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12181.246976] LNet: Removed LNI 192.168.204.158@tcp [12182.004515] Key type .llcrypt unregistered [12182.007089] Key type ._llcrypt unregistered [12212.803523] Key type ._llcrypt registered [12212.805353] Key type .llcrypt registered [12212.909676] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [12228.711204] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [12230.371426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12230.394724] alg: No test for adler32 (adler32-zlib) [12231.601333] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [12231.843365] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [12233.584151] Key type lgssc registered [12234.783806] Lustre: Echo OBD driver; http://www.lustre.org/ [12274.205673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12274.237517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12275.718663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12275.780856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12276.058807] Lustre: lustre-MDT0000: new disk, initializing [12276.236450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12276.310030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12282.877828] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12298.879213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12299.082227] Lustre: 319106:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12299.204946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12299.212779] Lustre: Skipped 1 previous similar message [12299.317139] Lustre: lustre-MDT0001: new disk, initializing [12299.473410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12299.526919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12299.547195] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12304.102754] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12313.784865] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12318.970988] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12325.684481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12325.902826] Lustre: lustre-OST0000: new disk, initializing [12325.907612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12325.921884] Lustre: 321079:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12326.025893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12332.342823] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12335.683248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12335.697598] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12335.840917] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12341.722227] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12343.636698] Lustre: Mounted lustre-client [12344.472489] Lustre: Unmounted lustre-client [12356.580120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12356.589759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12356.592831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12356.602065] Lustre: Skipped 1 previous similar message [12359.152956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12359.164699] Lustre: Skipped 2 previous similar messages [12360.585872] Lustre: server umount lustre-OST0000 complete [12364.257993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12364.259729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12364.277791] Lustre: Skipped 1 previous similar message [12364.286947] Lustre: Skipped 1 previous similar message [12369.377857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12369.384471] Lustre: Skipped 1 previous similar message [12370.514932] Lustre: server umount lustre-MDT0000 complete [12374.406305] LustreError: 319100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781299870 with bad export cookie 8227063688795518477 [12374.416586] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12374.419222] LustreError: 319100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12374.914562] Lustre: server umount lustre-MDT0001 complete [12386.069687] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [12388.648176] Key type lgssc unregistered [12401.250484] LNet: 323239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12401.262293] LNetError: 323239:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12473.974727] LNet: Removed LNI 192.168.204.158@tcp [12474.952505] Key type .llcrypt unregistered [12474.961087] Key type ._llcrypt unregistered [12496.263001] Key type ._llcrypt registered [12496.266379] Key type .llcrypt registered [12496.412883] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [12512.334801] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [12513.270386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12513.298830] alg: No test for adler32 (adler32-zlib) [12514.333846] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [12514.579326] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [12516.259281] Key type lgssc registered [12517.242126] Lustre: Echo OBD driver; http://www.lustre.org/ [12544.606656] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 17:33:58 (1781300038) [12550.537296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12550.560546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12551.887828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12551.928406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12552.025452] Lustre: lustre-MDT0000: new disk, initializing [12552.117239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12552.135940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12556.946901] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12568.250687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12568.410778] Lustre: 326467:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12568.473351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12568.479023] Lustre: Skipped 1 previous similar message [12568.582363] Lustre: lustre-MDT0001: new disk, initializing [12568.699312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12568.745292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12568.764127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12573.225273] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12583.123392] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12588.726376] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12594.686929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12595.038164] Lustre: lustre-OST0000: new disk, initializing [12595.041729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12595.052511] Lustre: 328432:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12595.129453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12601.755397] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12604.977875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12604.985078] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12605.072619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12611.249131] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12630.501514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12630.525115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12632.833833] Lustre: server umount lustre-OST0000 complete [12640.230706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12640.244331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12640.263959] Lustre: Skipped 1 previous similar message [12640.272869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12640.279954] Lustre: Skipped 1 previous similar message [12643.247733] Lustre: server umount lustre-MDT0000 complete [12646.977724] LustreError: 326459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781300142 with bad export cookie 9484410627269304801 [12646.994923] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12647.000378] LustreError: 326459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12647.429664] Lustre: server umount lustre-MDT0001 complete [12659.679197] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [12662.460755] Key type lgssc unregistered [12662.776250] LNet: 330341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12662.780982] LNetError: 330341:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12663.850595] LNet: Removed LNI 192.168.204.158@tcp [12664.626860] Key type .llcrypt unregistered [12664.628856] Key type ._llcrypt unregistered [12687.766559] Key type ._llcrypt registered [12687.768533] Key type .llcrypt registered [12687.902361] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [12689.004066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12689.568342] alg: No test for adler32 (adler32-zlib) [12690.661746] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [12690.939118] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [12692.664422] Key type lgssc registered [12693.991715] Lustre: Echo OBD driver; http://www.lustre.org/ [12702.469771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12702.497798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12704.117216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12709.629178] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12713.444845] LustreError: 331614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12713.466565] LustreError: 331614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12718.568800] LustreError: 331615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12719.246414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12719.716746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12724.527340] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12732.160325] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12737.614921] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12744.321966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12744.682134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12750.889683] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12758.590688] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12769.291399] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12771.066766] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12775.907772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12775.917625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12775.945431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12776.541966] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.58@tcp (stopping) [12779.494725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12779.503700] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12781.558316] Lustre: server umount lustre-MDT0000 complete [12784.613201] LustreError: 331615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12784.625748] LustreError: 331615:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12786.087862] LustreError: 331596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781300281 with bad export cookie 5406051351696994026 [12786.098471] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12786.108301] LustreError: 331596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12789.732775] LustreError: 331609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12789.739662] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12789.759288] LustreError: 331609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12789.789418] Lustre: Skipped 1 previous similar message [12789.802604] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12789.816341] Lustre: Skipped 3 previous similar messages [12792.496528] Lustre: server umount lustre-MDT0001 complete [12796.583516] Lustre: server umount lustre-OST0000 complete [12807.732749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12812.739590] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12816.274306] Lustre: 335793:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.158@tcp [12818.826974] Lustre: server umount MGS complete [12827.584775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12828.009151] LustreError: 336367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12828.030797] LustreError: 336367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12828.087780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12832.885203] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12837.357070] LustreError: 336368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12837.383601] LustreError: 336368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12841.481808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12841.879042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12845.942319] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12854.553189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12854.839145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12861.076916] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [12868.795455] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12870.380706] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12876.235437] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12878.248144] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12883.427915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12883.436623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12883.440550] Lustre: Skipped 1 previous similar message [12883.448724] Lustre: Skipped 2 previous similar messages [12885.585651] Lustre: server umount lustre-OST0000 complete [12893.166912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12893.175220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12893.186558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12893.194484] Lustre: Skipped 1 previous similar message [12896.081107] Lustre: server umount lustre-MDT0000 complete [12898.786655] LustreError: 336364:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12898.811885] LustreError: 336364:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12899.549481] LustreError: 336347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781300395 with bad export cookie 5406051351696994866 [12899.556377] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12899.560642] LustreError: 336347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12899.912761] Lustre: server umount lustre-MDT0001 complete [12910.031653] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [12912.797576] Key type lgssc unregistered [12913.101522] LNet: 339805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12913.119457] LNetError: 339805:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12913.142261] LNet: Removed LNI 192.168.204.158@tcp [12914.032689] Key type .llcrypt unregistered [12914.035417] Key type ._llcrypt unregistered [12932.798293] Key type ._llcrypt registered [12932.800522] Key type .llcrypt registered [12932.905808] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [12949.322799] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [12950.670021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12950.692659] alg: No test for adler32 (adler32-zlib) [12951.739778] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [12951.928597] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [12953.617543] Key type lgssc registered [12954.698692] Lustre: Echo OBD driver; http://www.lustre.org/ [12988.544340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12988.563951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12989.909922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12989.958400] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12990.059837] Lustre: lustre-MDT0000: new disk, initializing [12990.169886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12990.207575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12994.581619] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [13005.873742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13005.944521] Lustre: 342968:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13006.019239] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13006.023213] Lustre: Skipped 1 previous similar message [13006.100556] Lustre: lustre-MDT0001: new disk, initializing [13006.222885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13006.266060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13006.278183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13010.374795] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [13020.653919] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13027.115293] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13035.671896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13036.130736] Lustre: lustre-OST0000: new disk, initializing [13036.136952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13036.142628] Lustre: 344936:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13036.244046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13042.596734] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [13043.282177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13043.294326] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13043.391727] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13051.993644] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13056.933458] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13057.154105] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13063.175347] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13063.554846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13069.287050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13069.291169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13069.328213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13071.902508] Lustre: server umount lustre-OST0000 complete [13077.998226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13078.016778] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13078.033611] Lustre: Skipped 1 previous similar message [13078.045615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13078.054731] Lustre: Skipped 1 previous similar message [13079.522814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13079.524432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13079.552433] Lustre: Skipped 1 previous similar message [13082.381440] Lustre: server umount lustre-MDT0000 complete [13085.800663] LustreError: 343763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781300581 with bad export cookie 13496739637481376720 [13085.801056] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13085.821276] LustreError: 343763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13085.981728] Lustre: server umount lustre-MDT0001 complete [13095.648587] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 17:43:08 (1781300588) [13110.486937] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [13119.292190] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [13157.374074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13157.645551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13157.692337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13157.800405] Lustre: lustre-MDT0000: new disk, initializing [13157.890986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13157.915171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13162.075481] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [13172.704540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13172.804272] Lustre: 349697:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13172.852813] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13172.856416] Lustre: Skipped 1 previous similar message [13172.929919] Lustre: lustre-MDT0001: new disk, initializing [13173.011975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13173.047742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13173.059593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13177.216686] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [13188.097191] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13195.178225] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13202.498781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13202.876944] Lustre: lustre-OST0000: new disk, initializing [13202.887793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13202.896992] Lustre: 351667:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13203.054784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13209.059000] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [13209.164786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13209.187608] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13209.318746] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13218.230570] Lustre: DEBUG MARKER: oleg458-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13255.650371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13255.650769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13255.659860] LustreError: Skipped 1 previous similar message [13255.673446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13255.678168] Lustre: Skipped 1 previous similar message [13260.738769] Lustre: server umount lustre-OST0000 complete [13265.386441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13265.400986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13265.417443] Lustre: Skipped 1 previous similar message [13265.425035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13271.316270] Lustre: server umount lustre-MDT0000 complete [13274.877954] LustreError: 349689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781300770 with bad export cookie 13496739637481377602 [13274.888990] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13274.891937] LustreError: 349689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13275.154071] Lustre: server umount lustre-MDT0001 complete [13285.928607] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [13288.681456] Key type lgssc unregistered [13288.935594] LNet: 353840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13288.947928] LNetError: 353840:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13288.967590] LNet: Removed LNI 192.168.204.158@tcp [13289.710191] Key type .llcrypt unregistered [13289.712082] Key type ._llcrypt unregistered [13383.256505] Key type ._llcrypt registered [13383.258964] Key type .llcrypt registered [13383.381155] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [13383.722864] Key type .llcrypt unregistered [13383.725873] Key type ._llcrypt unregistered [13454.845409] Key type ._llcrypt registered [13454.849908] Key type .llcrypt registered [13454.986612] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [13455.339322] Key type .llcrypt unregistered [13455.343315] Key type ._llcrypt unregistered [13484.759696] Key type ._llcrypt registered [13484.763890] Key type .llcrypt registered [13484.860790] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [13485.178393] Key type .llcrypt unregistered [13485.182019] Key type ._llcrypt unregistered [13513.279288] Key type ._llcrypt registered [13513.281551] Key type .llcrypt registered [13513.387817] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [13513.719372] Key type .llcrypt unregistered [13513.721652] Key type ._llcrypt unregistered [13589.642484] Key type ._llcrypt registered [13589.645269] Key type .llcrypt registered [13589.737521] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [13589.972385] Key type .llcrypt unregistered [13589.976140] Key type ._llcrypt unregistered [13618.697539] Key type ._llcrypt registered [13618.704529] Key type .llcrypt registered [13618.810626] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [13619.157137] Key type .llcrypt unregistered [13619.159135] Key type ._llcrypt unregistered [13647.879433] Key type ._llcrypt registered [13647.884914] Key type .llcrypt registered [13648.009658] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [13648.269382] Key type .llcrypt unregistered [13648.272246] Key type ._llcrypt unregistered [13676.710682] Key type ._llcrypt registered [13676.714266] Key type .llcrypt registered [13676.791577] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [13676.991326] Key type .llcrypt unregistered [13676.995686] Key type ._llcrypt unregistered [13704.203795] Key type ._llcrypt registered [13704.207068] Key type .llcrypt registered [13704.317090] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [13704.846286] Key type .llcrypt unregistered [13704.855928] Key type ._llcrypt unregistered [13731.198890] Key type ._llcrypt registered [13731.202699] Key type .llcrypt registered [13731.316160] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [13731.600196] Key type .llcrypt unregistered [13731.601482] Key type ._llcrypt unregistered [13792.847195] Key type ._llcrypt registered [13792.851382] Key type .llcrypt registered [13792.977428] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [13793.233546] Key type .llcrypt unregistered [13793.236807] Key type ._llcrypt unregistered [13821.013775] Key type ._llcrypt registered [13821.016462] Key type .llcrypt registered [13821.117664] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [13821.359428] Key type .llcrypt unregistered [13821.362994] Key type ._llcrypt unregistered [13848.413298] Key type ._llcrypt registered [13848.416738] Key type .llcrypt registered [13848.531759] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [13848.816302] Key type .llcrypt unregistered [13848.818966] Key type ._llcrypt unregistered [13898.265638] Key type ._llcrypt registered [13898.272232] Key type .llcrypt registered [13898.399235] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [13914.363550] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [13915.453080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13915.480212] alg: No test for adler32 (adler32-zlib) [13916.743250] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [13916.982797] LNet: Added LNI 192.168.204.158@tcp [8/256/0/180] [13918.656331] Key type lgssc registered [13919.826606] Lustre: Echo OBD driver; http://www.lustre.org/ [13953.389564] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13953.411721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13954.741875] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13954.809325] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13954.914477] Lustre: tst250fs-MDT0000: new disk, initializing [13955.012347] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13955.034419] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13959.795715] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [13970.328338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13970.461319] Lustre: 374123:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13970.524643] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13970.527327] Lustre: Skipped 1 previous similar message [13970.625446] Lustre: tst250fs-MDT0001: new disk, initializing [13970.710044] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13970.740122] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13970.751865] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13976.490704] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [13983.403735] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13993.010366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13993.498708] Lustre: tst250fs-OST0000: new disk, initializing [13993.507338] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13993.515723] Lustre: 375757:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13993.681423] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13998.719891] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13998.741830] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13998.918270] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14001.047841] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [14012.132082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14012.293540] Lustre: tst250fs-OST0001: new disk, initializing [14012.296977] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14012.302895] Lustre: 376627:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14012.405425] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14018.932118] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_default_debug -1 all [14021.219069] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14021.225137] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14021.404090] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14025.069281] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14027.156860] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14027.166401] Lustre: Skipped 1 previous similar message [14047.205788] Lustre: tst250fs-MDT0000-lwp-OST0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14047.209917] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14047.223685] Lustre: Skipped 1 previous similar message [14047.237183] Lustre: Skipped 3 previous similar messages [14054.112353] Lustre: server umount tst250fs-MDT0000 complete [14057.445513] LustreError: 374131:0:(ldlm_lib.c:1179:target_handle_connect()) tst250fs-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14057.479815] LustreError: 374131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [14057.980499] LustreError: 374920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781301553 with bad export cookie 5861262388294755109 [14057.981908] LustreError: MGC192.168.204.158@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14057.987576] LustreError: 374920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14058.258462] Lustre: server umount tst250fs-MDT0001 complete [14071.940513] Lustre: server umount tst250fs-OST0000 complete [14086.017331] Lustre: server umount tst250fs-OST0001 complete [14105.804791] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [14116.360620] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [14145.876711] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 18:00:39 (1781301639) [14147.293799] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14162.675290] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing set_hostid [14173.467548] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing load_modules_local [14207.575642] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13924 sec ========= 18:01:40 (1781301700) [14209.564959] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:01:42 (1781301702) === [14213.430857] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:01:46 (1781301706) === [14256.518445] Lustre: DEBUG MARKER: oleg458-server.virtnet: executing unload_modules_local [14259.557772] Key type lgssc unregistered [14259.844423] LNet: 385768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14259.867774] LNetError: 385768:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14259.890534] LNet: Removed LNI 192.168.204.158@tcp [14260.799234] Key type .llcrypt unregistered [14260.803125] Key type ._llcrypt unregistered