[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 539624103 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002344] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006594] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009129] LSM: Security Framework initializing [ 0.010058] Yama: becoming mindful. [ 0.011046] SELinux: Initializing. [ 0.012074] *** VALIDATE selinux *** [ 0.020365] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024719] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027109] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028111] *** VALIDATE tmpfs *** [ 0.030373] *** VALIDATE proc *** [ 0.031253] *** VALIDATE cgroup *** [ 0.032008] *** VALIDATE cgroup2 *** [ 0.033260] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036025] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.040395] debug: unmapping init [mem 0xffffffff99859000-0xffffffff99860fff] [ 0.042868] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043654] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044027] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046014] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048017] ... max period: 00007fffffffffff [ 0.049014] ... fixed-purpose events: 3 [ 0.050011] ... event mask: 000000070000000f [ 0.051260] rcu: Hierarchical SRCU implementation. [ 0.053466] smp: Bringing up secondary CPUs ... [ 0.054567] x86: Booting SMP configuration: [ 0.055022] .... node #0, CPUs: #1 #2 #3 [ 0.059420] smp: Brought up 1 node, 4 CPUs [ 0.061017] smpboot: Max logical packages: 1 [ 0.062015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.245912] node 0 deferred pages initialised in 181ms [ 0.249223] devtmpfs: initialized [ 0.250260] x86/mm: Memory block size: 128MB [ 0.253266] gcov: version magic: 0x41383552 [ 0.256368] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.259097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.261447] pinctrl core: initialized pinctrl subsystem [ 0.263274] [ 0.263726] ************************************************************* [ 0.267018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.269017] ** ** [ 0.272022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.274031] ** ** [ 0.276020] ** This means that this kernel is built to expose internal ** [ 0.278017] ** IOMMU data structures, which may compromise security on ** [ 0.281027] ** your system. ** [ 0.283013] ** ** [ 0.285019] ** If you see this message and you are not debugging the ** [ 0.287015] ** kernel, report this immediately to your vendor! ** [ 0.289019] ** ** [ 0.291020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.293021] ************************************************************* [ 0.296885] NET: Registered protocol family 16 [ 0.298492] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.301087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.304084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.308054] cpuidle: using governor menu [ 0.310832] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.313837] PCI: Using configuration type 1 for base access [ 0.316165] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.330066] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.332021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.336238] cryptd: max_cpu_qlen set to 1000 [ 0.339275] ACPI: Added _OSI(Module Device) [ 0.340021] ACPI: Added _OSI(Processor Device) [ 0.341014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.342014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.347239] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.353537] ACPI: Interpreter enabled [ 0.355244] ACPI: PM: (supports S0 S3 S4 S5) [ 0.356014] ACPI: Using IOAPIC for interrupt routing [ 0.358147] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.361446] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.370967] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.373059] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.375028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.378126] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.384518] acpiphp: Slot [2] registered [ 0.386149] acpiphp: Slot [5] registered [ 0.387125] acpiphp: Slot [6] registered [ 0.388130] acpiphp: Slot [7] registered [ 0.389114] acpiphp: Slot [8] registered [ 0.391166] acpiphp: Slot [9] registered [ 0.392105] acpiphp: Slot [10] registered [ 0.393106] acpiphp: Slot [3] registered [ 0.394106] acpiphp: Slot [4] registered [ 0.396172] acpiphp: Slot [11] registered [ 0.397135] acpiphp: Slot [12] registered [ 0.399143] acpiphp: Slot [13] registered [ 0.400125] acpiphp: Slot [14] registered [ 0.402132] acpiphp: Slot [15] registered [ 0.403112] acpiphp: Slot [16] registered [ 0.404158] acpiphp: Slot [17] registered [ 0.405109] acpiphp: Slot [18] registered [ 0.407143] acpiphp: Slot [19] registered [ 0.408131] acpiphp: Slot [20] registered [ 0.410142] acpiphp: Slot [21] registered [ 0.411134] acpiphp: Slot [22] registered [ 0.413161] acpiphp: Slot [23] registered [ 0.414239] acpiphp: Slot [24] registered [ 0.416169] acpiphp: Slot [25] registered [ 0.418117] acpiphp: Slot [26] registered [ 0.419127] acpiphp: Slot [27] registered [ 0.421119] acpiphp: Slot [28] registered [ 0.422185] acpiphp: Slot [29] registered [ 0.424181] acpiphp: Slot [30] registered [ 0.426165] acpiphp: Slot [31] registered [ 0.428104] PCI host bridge to bus 0000:00 [ 0.429025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.432035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.434030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.437035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.440040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.443021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.444242] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.449110] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.452317] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.462020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.466916] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.469028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.472028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.477030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.481398] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.484099] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.487062] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.489874] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.496020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.507917] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.512073] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.520146] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.530021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.539025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.556027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.569517] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.576022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.583021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.604017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.614879] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.620022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.626025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.638023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.647422] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.652019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.658039] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.674025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.685184] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.695021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.702021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.719024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.731107] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.740023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.747021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.764026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.779384] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.782425] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.784411] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.788597] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.791272] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.797041] iommu: Default domain type: Passthrough [ 0.798626] SCSI subsystem initialized [ 0.800244] ACPI: bus type USB registered [ 0.802144] usbcore: registered new interface driver usbfs [ 0.804203] usbcore: registered new interface driver hub [ 0.806174] usbcore: registered new device driver usb [ 0.808274] pps_core: LinuxPPS API ver. 1 registered [ 0.810083] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.814086] PTP clock support registered [ 0.816086] EDAC MC: Ver: 3.0.0 [ 0.818167] PCI: Using ACPI for IRQ routing [ 0.821096] NetLabel: Initializing [ 0.822017] NetLabel: domain hash size = 128 [ 0.823013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.825138] NetLabel: unlabeled traffic allowed by default [ 0.828147] vgaarb: loaded [ 0.829363] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.831019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.838297] clocksource: Switched to clocksource kvm-clock [ 0.954381] VFS: Disk quotas dquot_6.6.0 [ 0.955951] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.959666] *** VALIDATE ramfs *** [ 0.961702] *** VALIDATE hugetlbfs *** [ 0.963749] pnp: PnP ACPI init [ 0.966826] pnp: PnP ACPI: found 6 devices [ 0.983823] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.987755] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.990189] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.992534] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.995173] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.998326] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.001711] NET: Registered protocol family 2 [ 1.004594] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.009251] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.013104] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.018166] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.021607] TCP: Hash tables configured (established 65536 bind 65536) [ 1.024717] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.027638] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.030655] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.033549] NET: Registered protocol family 1 [ 1.036208] RPC: Registered named UNIX socket transport module. [ 1.038658] RPC: Registered udp transport module. [ 1.040124] RPC: Registered tcp transport module. [ 1.041625] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.044781] NET: Registered protocol family 44 [ 1.046215] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.048128] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.050118] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.052104] PCI: CLS 0 bytes, default 64 [ 1.053536] Unpacking initramfs... [ 2.478726] debug: unmapping init [mem 0xffff8b5dbcc54000-0xffff8b5dbffbffff] [ 2.482957] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.485534] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.488790] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.992336] Initialise system trusted keyrings [ 2.994289] Key type blacklist registered [ 2.996800] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.009596] zbud: loaded [ 3.014092] *** VALIDATE nfs *** [ 3.015376] *** VALIDATE nfs4 *** [ 3.016992] pstore: using deflate compression [ 3.020118] Platform Keyring initialized [ 3.134478] NET: Registered protocol family 38 [ 3.137100] Key type asymmetric registered [ 3.139281] Asymmetric key parser 'x509' registered [ 3.141736] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.146522] io scheduler mq-deadline registered [ 3.148074] io scheduler kyber registered [ 3.149573] io scheduler bfq registered [ 3.151567] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.154648] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.157827] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.160984] ACPI: Power Button [PWRF] [ 3.166783] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.173434] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.185698] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.192117] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.205681] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.233184] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.260727] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.266123] Non-volatile memory driver v1.3 [ 3.268025] Linux agpgart interface v0.103 [ 3.302428] virtio_blk virtio1: [vda] 149896 512-byte logical blocks (76.7 MB/73.2 MiB) [ 3.305649] vda: detected capacity change from 0 to 76746752 [ 3.330111] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.333429] vdb: detected capacity change from 0 to 1073741824 [ 3.353474] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.356990] vdc: detected capacity change from 0 to 2621440000 [ 3.371781] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.375113] vdd: detected capacity change from 0 to 2621440000 [ 3.392291] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.395147] vde: detected capacity change from 0 to 4294967296 [ 3.410035] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.413052] vdf: detected capacity change from 0 to 4294967296 [ 3.424735] libphy: Fixed MDIO Bus: probed [ 3.431302] usbcore: registered new interface driver usbserial_generic [ 3.434031] usbserial: USB Serial support registered for generic [ 3.436533] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.441232] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.443263] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.447211] mousedev: PS/2 mouse device common for all mice [ 3.450487] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.454836] rtc_cmos 00:05: RTC can wake from S4 [ 3.456898] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.461799] rtc_cmos 00:05: registered as rtc0 [ 3.464072] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.464953] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.471585] intel_pstate: CPU model not supported [ 3.475655] hid: raw HID events driver (C) Jiri Kosina [ 3.479138] usbcore: registered new interface driver usbhid [ 3.481264] usbhid: USB HID core driver [ 3.482818] drop_monitor: Initializing network drop monitor service [ 3.485437] Initializing XFRM netlink socket [ 3.487453] NET: Registered protocol family 10 [ 3.490443] Segment Routing with IPv6 [ 3.492233] NET: Registered protocol family 17 [ 3.494434] mpls_gso: MPLS GSO support [ 3.500774] RAS: Correctable Errors collector initialized. [ 3.505138] AVX version of gcm_enc/dec engaged. [ 3.507155] AES CTR mode by8 optimization enabled [ 3.587697] sched_clock: Marking stable (3587674838, 0)->(4532353905, -944679067) [ 3.591626] registered taskstats version 1 [ 3.593869] Loading compiled-in X.509 certificates [ 3.596070] zswap: loaded using pool lzo/zbud [ 3.622361] Key type big_key registered [ 3.634395] Key type encrypted registered [ 3.636225] ima: No TPM chip found, activating TPM-bypass! [ 3.638272] ima: Allocated hash algorithm: sha1 [ 3.639594] ima: No architecture policies found [ 3.640850] evm: Initialising EVM extended attributes: [ 3.642283] evm: security.selinux [ 3.643187] evm: security.ima [ 3.643911] evm: security.capability [ 3.644893] evm: HMAC attrs: 0x1 [ 3.647283] rtc_cmos 00:05: setting system clock to 2026-09-03 08:00:57 UTC (1788422457) [ 3.652457] debug: unmapping init [mem 0xffffffff9a803000-0xffffffff9a9fffff] [ 3.654754] debug: unmapping init [mem 0xffffffff99582000-0xffffffff99858fff] [ 3.664066] Write protecting the kernel read-only data: 28672k [ 3.666893] debug: unmapping init [mem 0xffffffff97c03000-0xffffffff97dfffff] [ 3.668700] debug: unmapping init [mem 0xffffffff98514000-0xffffffff985fffff] [ 3.705936] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.716418] systemd[1]: Detected virtualization kvm. [ 3.718544] systemd[1]: Detected architecture x86-64. [ 3.721277] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.746481] systemd[1]: No hostname configured. [ 3.748563] systemd[1]: Set hostname to . [ 3.750985] random: systemd: uninitialized urandom read (16 bytes read) [ 3.753614] systemd[1]: Initializing machine ID from random generator. [ 3.800268] random: ln: uninitialized urandom read (6 bytes read) [ 3.878174] random: systemd: uninitialized urandom read (16 bytes read) [ 3.880959] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.888287] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.892649] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Timers. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.471693] device-mapper: uevent: version 1.0.3 [ 4.474492] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.199598] random: fast init done [ 5.223269] virtio_net virtio0 ens2: renamed from eth0 [ 5.321602] scsi host0: ata_piix [ 5.387201] scsi host1: ata_piix [ 5.389143] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.392048] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.999353] dracut-initqueue[583]: RTNETLINK answers: File exists [ 10.152114] random: crng init done [ 10.154413] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.534261] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.647500] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.912779] SELinux: Disabled at runtime. [ 11.976241] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.987310] systemd[1]: Detected virtualization kvm. [ 11.989187] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.500163] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.504124] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.509487] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.515388] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.519473] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.527896] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.537845] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. Startin[ 12.674734] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS g Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.978896] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.270654] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.299725] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.416964] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.427104] EDAC sbridge: Ver: 1.1.2 [ 15.013298] Key type dns_resolver registered [ 15.316468] NFS: Registering the id_resolver key type [ 15.321320] Key type id_resolver registered [ 15.323025] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg416-server login: [ 45.100279] libcfs: loading out-of-tree module taints kernel. [ 45.173625] Key type ._llcrypt registered [ 45.176042] Key type .llcrypt registered [ 45.310110] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 61.035674] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 62.439319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.457932] alg: No test for adler32 (adler32-zlib) [ 64.080080] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 65.369318] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 67.328764] Key type lgssc registered [ 69.850855] Lustre: Echo OBD driver; http://www.lustre.org/ [ 93.662906] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 131.841978] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 144.066110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 144.082724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 145.465823] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 145.517792] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 145.625443] Lustre: lustre-MDT0000: new disk, initializing [ 145.758468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 145.793238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 151.219100] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.587049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.693977] Lustre: 6503:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 167.753357] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 167.761141] Lustre: Skipped 1 previous similar message [ 167.913825] Lustre: lustre-MDT0001: new disk, initializing [ 168.054347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 168.104078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 168.117666] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 173.721436] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.861268] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 192.748325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 193.263744] Lustre: lustre-OST0000: new disk, initializing [ 193.269392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 193.280718] Lustre: 8446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 193.413732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 198.267918] hrtimer: interrupt took 4876111 ns [ 201.301088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 201.308876] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.321775] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 201.459353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 216.046501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.159162] Lustre: lustre-OST0001: new disk, initializing [ 216.167493] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 216.172156] Lustre: 9520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.257170] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 222.894541] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.295709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 223.310149] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 223.395931] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 236.204273] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 248.046848] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 253.887731] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing check_logdir /tmp/testlogs/ [ 262.871950] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing yml_node [ 268.508502] Lustre: DEBUG MARKER: Client: 2.17.58.7 [ 271.881542] Lustre: DEBUG MARKER: MDS: 2.17.58.7 [ 275.232669] Lustre: DEBUG MARKER: OSS: 2.17.58.7 [ 276.928812] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 04:05:28 EDT 2026 [ 297.220266] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 299.072901] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 316.385352] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 316.392355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.399635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 320.481496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 320.486277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.516496] Lustre: Skipped 2 previous similar messages [ 325.601480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 325.611597] Lustre: Skipped 6 previous similar messages [ 330.208250] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 330.444561] Lustre: server umount lustre-MDT0000 complete [ 333.766742] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788422787 with bad export cookie 6070993355713857817 [ 333.772231] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 333.780522] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 335.842506] LustreError: 6511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 335.843645] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 335.843931] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 335.871272] LustreError: 6511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 335.878783] Lustre: Skipped 1 previous similar message [ 338.849117] LustreError: 7461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 338.862514] LustreError: 7461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 343.973794] LustreError: 6510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.973849] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.995315] Lustre: Skipped 4 previous similar messages [ 348.128147] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 348.414664] Lustre: server umount lustre-MDT0001 complete [ 352.698452] Lustre: server umount lustre-OST0000 complete [ 356.648696] Lustre: server umount lustre-OST0001 complete [ 363.599593] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 375.105963] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 413.475465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.750825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 413.800684] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 413.953259] Lustre: lustre-MDT0000: new disk, initializing [ 414.088885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 414.123468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 418.761361] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 430.659523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 430.849076] Lustre: 15077:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 431.022860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 431.033681] Lustre: Skipped 1 previous similar message [ 431.161457] Lustre: lustre-MDT0001: new disk, initializing [ 431.330426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 431.385090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 431.399217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 436.879469] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 448.588230] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 455.767491] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 463.741409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 464.231691] Lustre: lustre-OST0000: new disk, initializing [ 464.241764] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 464.257617] Lustre: 17047:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 464.404597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.919299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 465.934215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 466.141453] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 471.915498] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 484.814851] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 492.022883] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 492.400717] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 498.351552] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 498.712388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 501.733192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 501.744687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.769064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 506.960742] Lustre: server umount lustre-OST0000 complete [ 513.526450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 513.545532] LustreError: Skipped 1 previous similar message [ 513.563022] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 513.586048] Lustre: Skipped 1 previous similar message [ 518.067543] Lustre: server umount lustre-MDT0000 complete [ 520.691235] LustreError: 15087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.724365] LustreError: 15087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 523.520626] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788422977 with bad export cookie 6070993355713859504 [ 523.521814] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 523.545382] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 524.292681] Lustre: server umount lustre-MDT0001 complete [ 531.476900] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:09:43 (1788422983) [ 551.327500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 551.756676] LustreError: 19921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.831507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.301468] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 562.151930] LustreError: 19921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.186664] LustreError: 19921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 565.550458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 571.239696] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 580.760258] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 587.556227] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 589.713149] Lustre: Failing over lustre-MDT0000 [ 590.274201] Lustre: server umount lustre-MDT0000 complete [ 590.822158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 590.834664] Lustre: Skipped 1 previous similar message [ 590.842821] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.869897] LustreError: 19917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 591.099940] LustreError: mgc192.168.204.116@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 591.105061] LustreError: 21490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423044 with bad export cookie 6070993355713860428 [ 591.112092] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.128430] Lustre: 21523:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 592.995847] Lustre: 21522:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 598.497649] Lustre: 21524:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 603.618278] Lustre: 21523:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 608.740062] Lustre: 21524:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 609.851728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.289943] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 611.236067] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 611.240535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 611.294893] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 615.323971] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 625.729488] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.MGC*.mgs_server_uuid 1475 0 [ 630.765023] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 630.783499] Lustre: Skipped 1 previous similar message [ 630.789044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 630.794373] Lustre: Skipped 7 previous similar messages [ 635.890161] LustreError: 20656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.936100] LustreError: 20656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 635.984808] Lustre: server umount lustre-MDT0000 complete [ 639.871774] LustreError: 19903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423093 with bad export cookie 6070993355713860995 [ 639.879424] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.887339] LustreError: 19903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 640.142302] Lustre: server umount lustre-MDT0001 complete [ 648.724224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 649.321647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 649.333595] Lustre: Skipped 2 previous similar messages [ 654.748371] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 665.792915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 672.413576] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 681.182829] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 686.906992] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 689.337471] Lustre: Failing over lustre-MDT0000 [ 689.592820] Lustre: server umount lustre-MDT0000 complete [ 690.403706] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 690.420828] Lustre: Skipped 2 previous similar messages [ 691.190178] LustreError: 23245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423145 with bad export cookie 6070993355713861429 [ 691.191608] LustreError: mgc192.168.204.116@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 691.201066] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 691.218594] Lustre: 24910:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 702.947196] LustreError: 23258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 702.964186] LustreError: 23258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 708.065248] Lustre: 24907:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 708.082645] Lustre: 24907:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 3 previous similar messages [ 709.098627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 709.556795] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 710.567943] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 710.576138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 710.595101] Lustre: Skipped 1 previous similar message [ 710.613628] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 714.918939] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 725.712929] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.MGC*.mgs_server_uuid 1475 0 [ 730.084473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 730.093177] Lustre: Skipped 2 previous similar messages [ 736.080500] Lustre: server umount lustre-MDT0000 complete [ 739.639314] LustreError: 23246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423193 with bad export cookie 6070993355713861996 [ 739.649866] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 739.652666] LustreError: 23246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 740.096639] Lustre: server umount lustre-MDT0001 complete [ 749.776700] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 04:13:21 (1788423201) [ 767.170505] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 777.411604] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 815.260144] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 824.586717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 824.752315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 824.771467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 824.830470] Lustre: lustre-MDT0000: new disk, initializing [ 824.885417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 824.889664] Lustre: Skipped 2 previous similar messages [ 824.902448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 828.870387] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 839.796915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 839.903316] Lustre: 29642:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 840.489259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 845.445206] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 854.904957] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 860.547811] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 868.800524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 869.445516] Lustre: 31613:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 870.717097] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 870.842777] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 870.908129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 870.931747] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 879.522561] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 892.640098] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 899.053943] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 922.092184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 922.095316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.105809] LustreError: Skipped 1 previous similar message [ 922.109266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 922.122883] Lustre: Skipped 3 previous similar messages [ 922.148438] Lustre: Skipped 3 previous similar messages [ 925.914960] Lustre: server umount lustre-OST0000 complete [ 936.657519] Lustre: server umount lustre-MDT0000 complete [ 937.454620] LustreError: 29656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 937.495212] LustreError: 29656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 940.816306] LustreError: 29635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423394 with bad export cookie 6070993355713862423 [ 940.831528] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 940.835638] LustreError: 29635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 941.310615] Lustre: server umount lustre-MDT0001 complete [ 953.278748] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 956.350422] Key type lgssc unregistered [ 956.745810] LNet: 33590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.755167] LNetError: 33590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.781812] LNet: Removed LNI 192.168.204.116@tcp [ 958.079264] Key type .llcrypt unregistered [ 958.085330] Key type ._llcrypt unregistered [ 984.862340] Key type ._llcrypt registered [ 984.866930] Key type .llcrypt registered [ 985.070355] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 986.493271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 986.554821] alg: No test for adler32 (adler32-zlib) [ 987.593654] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 987.875486] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 989.577777] Key type lgssc registered [ 990.864882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 999.646510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 999.678037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.302774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.311672] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1011.681718] LustreError: 35002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1011.700107] LustreError: 35002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1016.622956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1016.808728] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1016.992791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1021.301559] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1028.733560] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1034.531755] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1042.406695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1043.046334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1047.222803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1050.346771] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1061.572872] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1062.715938] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1077.350338] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 04:18:49 (1788423529) [ 1079.395397] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1081.505706] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1089.407747] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 04:19:00 (1788423540) [ 1099.886487] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 04:19:11 (1788423551) [ 1101.054448] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1110.695670] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1110.699491] Lustre: Skipped 2 previous similar messages [ 1112.275273] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 04:19:24 (1788423564) [ 1116.062243] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1116.065656] Lustre: Skipped 3 previous similar messages [ 1135.246346] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1135.251316] Lustre: Skipped 6 previous similar messages [ 1138.213273] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 04:19:50 (1788423590) [ 1141.258178] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1145.037855] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1155.635596] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 04:20:07 (1788423607) [ 1157.673737] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1157.678159] Lustre: Skipped 3 previous similar messages [ 1172.798892] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 04:20:24 (1788423624) [ 1189.862510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1189.878080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1189.886529] Lustre: Skipped 1 previous similar message [ 1193.268587] Lustre: server umount lustre-OST0000 complete [ 1200.105860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.107920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1200.132667] Lustre: Skipped 2 previous similar messages [ 1200.169136] Lustre: Skipped 1 previous similar message [ 1205.220332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1205.228518] Lustre: Skipped 1 previous similar message [ 1209.316220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1209.321862] Lustre: Skipped 1 previous similar message [ 1211.360150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1211.737499] Lustre: server umount lustre-MDT0000 complete [ 1214.435079] LustreError: 35002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1214.461951] LustreError: 35002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1217.106923] LustreError: 35735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423670 with bad export cookie 17815807848068199739 [ 1217.116274] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1217.127266] LustreError: 35735:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1217.834820] Lustre: server umount lustre-MDT0001 complete [ 1228.713577] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 1232.026438] Key type lgssc unregistered [ 1232.302813] LNet: 42860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1232.310774] LNetError: 42860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1232.326730] LNet: Removed LNI 192.168.204.116@tcp [ 1233.253306] Key type .llcrypt unregistered [ 1233.257286] Key type ._llcrypt unregistered [ 1250.662176] Key type ._llcrypt registered [ 1250.665407] Key type .llcrypt registered [ 1250.789442] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 1267.363733] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 1268.476827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1268.508769] alg: No test for adler32 (adler32-zlib) [ 1269.564956] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 1269.892711] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 1271.560158] Key type lgssc registered [ 1272.975482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1320.171652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1320.186537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1321.614443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1321.670286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1321.773176] Lustre: lustre-MDT0000: new disk, initializing [ 1321.859862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1321.895166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1327.201736] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1338.328512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.452950] Lustre: 46010:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1338.547644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1338.553264] Lustre: Skipped 1 previous similar message [ 1338.631161] Lustre: lustre-MDT0001: new disk, initializing [ 1338.752365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1338.809582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1338.816515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1343.871479] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1355.297228] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1362.454471] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1370.161082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1370.418792] Lustre: lustre-OST0000: new disk, initializing [ 1370.422610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1370.434372] Lustre: 47981:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1370.581367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1377.544610] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1377.963494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1377.976455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1378.149824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1387.188565] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1391.589172] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1391.879274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1396.876430] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1397.211624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1403.879731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1403.882688] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1403.888180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1403.891480] Lustre: Skipped 1 previous similar message [ 1405.559191] Lustre: server umount lustre-OST0000 complete [ 1410.529407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1410.545330] LustreError: Skipped 1 previous similar message [ 1410.552342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1410.584570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1410.589772] Lustre: Skipped 1 previous similar message [ 1414.119157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1414.127438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1415.797156] Lustre: server umount lustre-MDT0000 complete [ 1420.089513] LustreError: 46003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423873 with bad export cookie 8396399572255144775 [ 1420.093531] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1420.105724] LustreError: 46003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1420.786778] Lustre: server umount lustre-MDT0001 complete [ 1432.200667] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 04:24:43 (1788423883) [ 1438.651055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1438.976888] LustreError: 50441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1438.993422] LustreError: 50441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1439.045031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1443.420506] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1444.330642] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1444.348988] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1448.418822] LustreError: 50441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1451.840122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.110614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1456.277856] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1463.859133] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1469.436100] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1476.452894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1476.875731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1483.579764] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1492.951954] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1500.854908] LustreError: 52970:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1501.740993] Lustre: Modifying parameter general.timeout=1 in log params [ 1503.555797] Lustre: Modifying parameter general.timeout=3 in log params [ 1503.563531] Lustre: Skipped 1 previous similar message [ 1506.264747] Lustre: Modifying parameter general.timeout=6 in log params [ 1506.268371] Lustre: Skipped 2 previous similar messages [ 1510.783809] Lustre: Modifying parameter general.timeout=11 in log params [ 1510.791268] Lustre: Skipped 4 previous similar messages [ 1519.313719] Lustre: Modifying parameter general.timeout=20 in log params [ 1519.322849] Lustre: Skipped 8 previous similar messages [ 1536.484109] Lustre: Modifying parameter general.timeout=35 in log params [ 1536.517352] Lustre: Skipped 14 previous similar messages [ 1569.578607] Lustre: Modifying parameter general.timeout=63 in log params [ 1569.586967] Lustre: Skipped 27 previous similar messages [ 1633.830555] Lustre: Modifying parameter general.timeout=125 in log params [ 1633.836533] Lustre: Skipped 61 previous similar messages [ 1651.357201] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 04:28:23 (1788424103) [ 1654.759542] Lustre: 59920:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1656.689503] Lustre: 60018:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1658.886398] Lustre: 60117:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1663.034971] Lustre: 60331:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 (mode = 2) failed: rc = -17 [ 1663.041748] Lustre: 60331:0:(mgs_llog.c:1456:mgs_modify_param()) Skipped 1 previous similar message [ 1697.877674] Lustre: 61793:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1697.893991] Lustre: 61793:0:(mgs_llog.c:1456:mgs_modify_param()) Skipped 1 previous similar message [ 1697.941038] LustreError: 61793:0:(mgs_handler.c:1215:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1704.091550] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 04:29:15 (1788424155) [ 1721.828741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1721.835324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1721.845179] Lustre: Skipped 2 previous similar messages [ 1721.888665] Lustre: Skipped 3 previous similar messages [ 1725.032821] Lustre: server umount lustre-MDT0000 complete [ 1726.969263] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1726.991450] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1730.457080] LustreError: 55600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788424184 with bad export cookie 8396399572255145692 [ 1730.482567] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1730.490546] LustreError: 55600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1731.066967] Lustre: server umount lustre-MDT0001 complete [ 1746.077670] Lustre: server umount lustre-OST0000 complete [ 1780.507754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1781.285797] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1781.372920] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1781.390261] Lustre: Skipped 27 previous similar messages [ 1781.880117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1787.450081] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1796.993628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.161738] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1797.241276] Lustre: 64325:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1797.352802] LustreError: 64337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1803.380622] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1811.506861] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1818.449396] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1827.837264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1828.166352] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1829.020101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1829.037751] Lustre: Skipped 1 previous similar message [ 1838.262100] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1847.628168] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1850.004069] Lustre: 66894:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1859.554090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.567222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1859.581812] Lustre: Skipped 1 previous similar message [ 1859.610634] Lustre: Skipped 1 previous similar message [ 1864.475588] Lustre: server umount lustre-OST0000 complete [ 1869.418221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1869.453934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1874.405100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1876.222469] Lustre: server umount lustre-MDT0000 complete [ 1880.035852] LustreError: 64337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1880.053623] LustreError: 64337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1880.303847] LustreError: 66199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788424334 with bad export cookie 8396399572255148912 [ 1880.304661] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1880.316413] LustreError: 66199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1880.733598] Lustre: server umount lustre-MDT0001 complete [ 1896.700625] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 1900.151556] Key type lgssc unregistered [ 1900.560541] LNet: 68106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.583986] LNetError: 68106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1900.608917] LNet: Removed LNI 192.168.204.116@tcp [ 1901.855479] Key type .llcrypt unregistered [ 1901.857768] Key type ._llcrypt unregistered [ 1929.020252] Key type ._llcrypt registered [ 1929.022539] Key type .llcrypt registered [ 1929.135259] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 1930.426068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1930.600048] alg: No test for adler32 (adler32-zlib) [ 1931.873490] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 1932.133258] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 1933.808476] Key type lgssc registered [ 1935.442370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1946.425797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1946.437654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.140838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1953.699692] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1957.368565] LustreError: 69518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1957.391381] LustreError: 69518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1962.468173] LustreError: 69519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1966.534627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1967.738631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1975.371631] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 1983.762409] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1991.302139] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2001.616398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2003.127157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2015.366580] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2025.737467] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2034.443696] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2048.995588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2049.011963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.049754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2049.525418] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.557784] Lustre: Skipped 1 previous similar message [ 2049.567249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2049.570816] Lustre: Skipped 2 previous similar messages [ 2051.965456] Lustre: server umount lustre-MDT0000 complete [ 2054.643765] LustreError: 70443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2054.668760] LustreError: 70443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2056.709168] LustreError: 69499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788424510 with bad export cookie 14373608240453491326 [ 2056.721950] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2056.740885] LustreError: 69499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2057.588186] Lustre: server umount lustre-MDT0001 complete [ 2073.073649] Lustre: server umount lustre-OST0000 complete [ 2115.714817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2116.241497] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2116.278558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2116.958757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2123.855949] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2134.542709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.594752] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2134.631175] Lustre: 74398:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2135.308726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2141.054852] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2149.831732] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2156.234095] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2163.508776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2163.712860] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2164.062680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.055792] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2182.737348] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2185.263834] Lustre: 76977:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2187.068524] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2196.470132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2196.474189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2196.475803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2196.504429] Lustre: Skipped 1 previous similar message [ 2199.769302] Lustre: server umount lustre-OST0000 complete [ 2206.691807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.696429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2206.716617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2206.716630] Lustre: Skipped 1 previous similar message [ 2206.753417] LustreError: Skipped 1 previous similar message [ 2210.830355] Lustre: server umount lustre-MDT0000 complete [ 2211.820592] LustreError: 76100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2211.860793] LustreError: 76100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2215.514589] LustreError: 74392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788424669 with bad export cookie 14373608240453492229 [ 2215.515321] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2215.531644] LustreError: 74392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2215.978743] Lustre: server umount lustre-MDT0001 complete [ 2227.762477] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2230.721233] Key type lgssc unregistered [ 2231.054136] LNet: 78186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2231.064457] LNetError: 78186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2231.084147] LNet: Removed LNI 192.168.204.116@tcp [ 2232.047416] Key type .llcrypt unregistered [ 2232.050370] Key type ._llcrypt unregistered [ 2260.133983] Key type ._llcrypt registered [ 2260.136475] Key type .llcrypt registered [ 2260.219445] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2261.133690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2261.251859] alg: No test for adler32 (adler32-zlib) [ 2262.454201] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 2262.866402] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2264.768256] Key type lgssc registered [ 2266.773365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2278.348230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2278.369403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.311583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2286.612395] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2290.661234] LustreError: 79596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2290.675299] LustreError: 79596:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2295.782073] LustreError: 79597:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2296.136354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2296.908817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2301.940985] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2311.155948] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2317.713618] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2324.846521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2325.510986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2331.974547] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2340.819392] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2347.869884] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2349.048992] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2350.159184] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2352.264914] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2352.268207] Lustre: Skipped 1 previous similar message [ 2356.376940] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2356.384477] Lustre: Skipped 3 previous similar messages [ 2364.568541] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2364.584311] Lustre: Skipped 7 previous similar messages [ 2381.120955] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2381.130489] Lustre: Skipped 17 previous similar messages [ 2413.837408] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2413.844833] Lustre: Skipped 35 previous similar messages [ 2449.809900] Lustre: 87700:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2452.412861] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 04:41:44 (1788424904) [ 2454.041193] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2456.371217] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 04:41:48 (1788424908) [ 2463.721249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2463.733307] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2463.763308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2464.737355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2464.738266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2466.508745] Lustre: server umount lustre-OST0000 complete [ 2474.985566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2475.004441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2475.016072] Lustre: Skipped 1 previous similar message [ 2475.020963] Lustre: Skipped 1 previous similar message [ 2480.097734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2480.111813] Lustre: Skipped 2 previous similar messages [ 2484.195110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2484.207405] Lustre: Skipped 1 previous similar message [ 2486.240900] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2486.404572] Lustre: server umount lustre-MDT0000 complete [ 2489.313729] LustreError: 79592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.344135] LustreError: 79592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2491.068563] LustreError: 79577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788424944 with bad export cookie 7451577387523547296 [ 2491.072267] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2491.095845] LustreError: 79577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2491.596630] Lustre: server umount lustre-MDT0001 complete [ 2504.426416] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2509.090442] Key type lgssc unregistered [ 2509.448480] LNet: 88949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2509.456228] LNetError: 88949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2509.471057] LNet: Removed LNI 192.168.204.116@tcp [ 2510.660465] Key type .llcrypt unregistered [ 2510.662053] Key type ._llcrypt unregistered [ 2514.210070] Key type ._llcrypt registered [ 2514.211750] Key type .llcrypt registered [ 2514.301904] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2520.015435] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules [ 2521.404873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2521.527332] alg: No test for adler32 (adler32-zlib) [ 2522.679674] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 2522.975535] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2524.665370] Key type lgssc registered [ 2525.918954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2527.134423] LustreError: 90183:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2544.132176] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2548.150394] LustreError: 90183:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2553.809058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2553.830187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2555.420961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2560.062162] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2565.604380] LustreError: 91178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2565.632639] LustreError: 91178:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2570.219919] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 04:43:41 (1788425021) [ 2570.724704] LustreError: 91179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2575.184593] LustreError: 91210:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2575.193849] LustreError: 91210:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2575.203303] LustreError: 91210:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2575.548241] Lustre: server umount lustre-MDT0000 complete [ 2590.137680] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 2592.988340] Key type lgssc unregistered [ 2593.335058] LNet: 92703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2593.341219] LNetError: 92703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2593.365593] LNet: Removed LNI 192.168.204.116@tcp [ 2594.188479] Key type .llcrypt unregistered [ 2594.193108] Key type ._llcrypt unregistered [ 2613.101447] Key type ._llcrypt registered [ 2613.107760] Key type .llcrypt registered [ 2613.257462] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2614.387208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2614.483083] alg: No test for adler32 (adler32-zlib) [ 2615.569484] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 2615.857610] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 2617.544184] Key type lgssc registered [ 2618.740700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2626.019791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2626.034653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.700702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.744394] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2637.292511] LustreError: 93921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2637.310920] LustreError: 93921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2642.091924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2642.900023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2648.050349] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2656.229905] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2662.703099] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2670.512331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2671.087826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2677.664177] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2686.158748] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2738.334816] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 04:46:30 (1788425190) [ 2740.393524] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2742.945663] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 04:46:34 (1788425194) [ 2749.923487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2749.927822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2749.937854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2749.962542] Lustre: Skipped 1 previous similar message [ 2749.979874] Lustre: Skipped 2 previous similar messages [ 2753.632163] Lustre: server umount lustre-MDT0000 complete [ 2755.048362] LustreError: 93915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.079042] LustreError: 93915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2757.350786] LustreError: 97079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425211 with bad export cookie 15793377957869594012 [ 2757.363359] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2757.366426] LustreError: 97079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2757.727983] Lustre: server umount lustre-MDT0001 complete [ 2770.640678] Lustre: server umount lustre-OST0000 complete [ 2781.988312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.580317] LustreError: 98196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2782.737667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2787.545628] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2787.813269] LustreError: 98197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2792.943528] LustreError: 98196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2796.429518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2797.249020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2802.108344] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2810.485591] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2816.496488] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2830.729859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2830.952401] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2830.978223] LustreError: 98184:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2831.022991] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2831.038339] LustreError: 100279:0:(tgt_mount.c:2799:server_fill_super()) Unable to start targets: -98 [ 2831.053221] LustreError: 100279:0:(tgt_mount.c:2249:server_put_super()) no obd lustre-OST0000 [ 2831.059501] LustreError: 100279:0:(tgt_mount.c:130:server_deregister_mount()) lustre-OST0000 not registered [ 2831.138173] Lustre: server umount lustre-OST0000 complete [ 2831.144381] LustreError: 100279:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2840.789311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2840.908175] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2840.915237] LustreError: 98184:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2840.937065] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2840.944725] LustreError: 100747:0:(tgt_mount.c:2799:server_fill_super()) Unable to start targets: -98 [ 2840.951266] LustreError: 100747:0:(tgt_mount.c:2249:server_put_super()) no obd lustre-OST0000 [ 2840.957040] LustreError: 100747:0:(tgt_mount.c:130:server_deregister_mount()) lustre-OST0000 not registered [ 2841.010938] Lustre: server umount lustre-OST0000 complete [ 2841.013761] LustreError: 100747:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2852.274450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2852.464073] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2852.469816] Lustre: Found index 0 for lustre-OST0000, updating log [ 2852.475089] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2852.594750] Lustre: lustre-OST0000: new disk, initializing [ 2852.604481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2852.614524] Lustre: 101281:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2852.948187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2859.984885] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2864.166506] Lustre: Failing over lustre-OST0000 [ 2864.485353] Lustre: server umount lustre-OST0000 complete [ 2868.705844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2868.711334] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2868.730542] Lustre: Skipped 1 previous similar message [ 2868.748083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2871.266872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2871.276546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2871.303467] Lustre: Skipped 1 previous similar message [ 2876.389636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2876.401837] Lustre: Skipped 1 previous similar message [ 2878.500782] Lustre: server umount lustre-MDT0000 complete [ 2881.508901] LustreError: 100354:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2881.537041] LustreError: 100354:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2882.964815] LustreError: 100299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425336 with bad export cookie 15793377957869596812 [ 2882.972226] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2882.980795] LustreError: 100299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2883.295614] Lustre: server umount lustre-MDT0001 complete [ 2892.280270] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 04:49:04 (1788425344) [ 2908.327725] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 2918.766223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2919.153016] LustreError: 103627:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2919.324732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.774904] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2934.237705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2939.572399] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2942.801439] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2949.094245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2949.685377] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2956.699761] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2959.907483] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2959.920182] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2959.947416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2964.974116] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2964.995835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2965.996457] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:03 [ 2966.048693] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2966.049018] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2966.072866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2966.545190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2966.638172] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2966.797938] Lustre: lustre-OST0001: new disk, initializing [ 2966.802636] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2966.825734] Lustre: 105950:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2967.254254] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2967.259707] Lustre: Skipped 2 previous similar messages [ 2972.893278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2972.902737] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2973.065762] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2973.774677] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 2982.526041] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2988.183481] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2993.633880] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2993.645401] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2993.654444] Lustre: Skipped 1 previous similar message [ 2993.671435] Lustre: Skipped 2 previous similar messages [ 2994.442295] Lustre: server umount lustre-MDT0001 complete [ 2997.369182] LustreError: 103624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2997.386778] LustreError: 103624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3004.107514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3004.234875] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3004.244309] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3004.293556] Lustre: 103615:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3005.770374] LustreError: 107508:0:(obd_mount.c:312:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3005.777150] LustreError: 107508:0:(tgt_mount.c:739:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3005.787353] LustreError: 103619:0:(mgc_request.c:718:do_requeue()) failed processing log: -17 [ 3005.993165] LustreError: 107536:0:(obd_config.c:2100:class_config_llog_handler()) mgc192.168.204.116@tcp: cfg command failed: rc = -17 [ 3006.000517] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3006.000517] [ 3009.415433] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3010.036979] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3018.856884] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 04:51:10 (1788425470) [ 3038.080935] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3054.529724] Lustre: 109572:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3080.256784] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3084.278890] Lustre: 110707:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3173.864946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3173.865955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3173.869384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3173.877785] Lustre: Skipped 3 previous similar messages [ 3173.908935] LustreError: Skipped 1 previous similar message [ 3179.807598] Lustre: server umount lustre-MDT0000 complete [ 3183.221539] LustreError: 105949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425637 with bad export cookie 15793377957869597897 [ 3183.227949] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3183.235614] LustreError: 105949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3183.471777] Lustre: server umount lustre-MDT0001 complete [ 3197.079265] Lustre: server umount lustre-OST0000 complete [ 3199.248133] Lustre: 93232:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788425637/real 1788425637] req@ffff8b5d087afb80 x1875299605550336/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788425653 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3199.267964] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3199.274887] Lustre: Skipped 1 previous similar message [ 3200.369436] Lustre: server umount lustre-OST0001 complete [ 3206.480521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3217.196812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3226.300100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3237.513579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3257.258693] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3269.435039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.491154] Lustre: lustre-MDT0000: reset Object Index mappings [ 3270.377814] LustreError: 114658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3270.399957] LustreError: 114658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3270.506852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.519104] Lustre: Skipped 1 previous similar message [ 3276.871498] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3287.866581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3287.888684] Lustre: lustre-MDT0001: reset Object Index mappings [ 3294.940632] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3298.750589] Lustre: 115810:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3306.137507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3312.344762] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3313.982567] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 3316.029232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:381 to 0x280000401:418) [ 3321.214801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3327.503928] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 3327.511122] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3328.567590] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3337.813651] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3342.406084] Lustre: 117685:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3393.957796] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 04:57:26 (1788425846) [ 3421.677667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3421.688101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3421.715091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3421.726253] Lustre: Skipped 7 previous similar messages [ 3427.046551] Lustre: server umount lustre-MDT0000 complete [ 3429.866992] LustreError: 115397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.892200] LustreError: 115397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 3430.930818] LustreError: 114640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425884 with bad export cookie 15793377957869645217 [ 3430.943889] LustreError: 114640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3430.946680] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3437.477835] Lustre: server umount lustre-MDT0001 complete [ 3441.279608] LustreError: 118448:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0x174:0x0].0x0 (ffff8b5d07f45d00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3447.557535] Lustre: server umount lustre-OST0000 complete [ 3457.843604] Lustre: server umount lustre-OST0001 complete [ 3464.960872] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 3473.754132] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3512.873822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3513.095550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3513.140527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3513.229614] Lustre: lustre-MDT0000: new disk, initializing [ 3513.309714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3513.318288] Lustre: Skipped 3 previous similar messages [ 3513.330505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3518.302685] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3525.333789] Lustre: server umount lustre-MDT0000 complete [ 3534.989723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3535.250273] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3535.263409] Lustre: 121959:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3535.272526] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3539.810345] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3546.603696] Lustre: server umount lustre-MDT0000 complete [ 3555.238614] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 05:00:07 (1788426007) [ 3557.051992] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3558.754808] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 05:00:10 (1788426010) [ 3577.619249] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 3589.352973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3596.000434] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3606.051741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3606.160256] Lustre: 124013:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3606.260841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3606.265854] Lustre: Skipped 1 previous similar message [ 3606.325539] Lustre: lustre-MDT0001: new disk, initializing [ 3606.447557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3606.485287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3611.778622] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3616.808804] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3623.746394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3624.035301] Lustre: lustre-OST0000: new disk, initializing [ 3624.043304] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3624.051256] Lustre: 125603:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3625.537500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3625.552880] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3625.651552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3631.710377] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3642.730189] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3642.915973] Lustre: lustre-OST0001: new disk, initializing [ 3642.920094] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3642.929912] Lustre: 126472:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3644.577930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3644.583735] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3644.755221] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3649.595824] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 3659.443544] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3663.632523] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4096.519975] Lustre: 132893:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4096.533705] Lustre: 132893:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4096.538751] Lustre: 132893:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4096.542534] Lustre: 132893:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4096.551454] Lustre: 132893:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4096.564625] Lustre: 132893:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4473.021975] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:15:23 (1788426923) [ 4489.697937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4489.721424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4489.726044] Lustre: Skipped 7 previous similar messages [ 4489.732586] Lustre: Skipped 8 previous similar messages [ 4494.818362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4494.837340] Lustre: Skipped 6 previous similar messages [ 4496.253461] Lustre: server umount lustre-MDT0000 complete [ 4499.959769] LustreError: 127299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4500.018526] LustreError: 127299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4501.906381] LustreError: 132920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788426955 with bad export cookie 15793377957869714923 [ 4501.959212] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4505.058735] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4505.079333] Lustre: Skipped 1 previous similar message [ 4505.098511] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4505.107311] Lustre: Skipped 1 previous similar message [ 4508.939500] Lustre: server umount lustre-MDT0001 complete [ 4520.423565] Lustre: server umount lustre-OST0000 complete [ 4530.724911] Lustre: server umount lustre-OST0001 complete [ 4537.773850] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 4548.510132] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 4592.348680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4592.611889] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4592.658878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4592.748651] Lustre: lustre-MDT0000: new disk, initializing [ 4592.860893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4592.866392] Lustre: Skipped 5 previous similar messages [ 4592.878884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4597.661043] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4608.903267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4608.959947] Lustre: 135973:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4609.029591] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4609.033285] Lustre: Skipped 1 previous similar message [ 4609.089953] Lustre: lustre-MDT0001: new disk, initializing [ 4609.227025] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4609.250447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4614.664500] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4626.648027] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4637.679334] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4647.772121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4648.187818] Lustre: lustre-OST0000: new disk, initializing [ 4648.198494] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4648.204625] Lustre: 137947:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4648.307737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4648.314539] Lustre: Skipped 1 previous similar message [ 4649.605707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4649.625071] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4649.887898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4656.776468] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 4668.725612] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4682.620455] Lustre: lustre-MDD0000: changelog on [ 4686.320120] Lustre: lustre-MDD0001: changelog on [ 6942.993207] Lustre: lustre-MDD0001: changelog off [ 6944.949150] Lustre: 135982:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6945.934996] Lustre: lustre-MDD0000: changelog off [ 6953.959053] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6953.962698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6953.964482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6953.964490] Lustre: Skipped 2 previous similar messages [ 6953.994525] Lustre: Skipped 2 previous similar messages [ 6959.082390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6959.089874] Lustre: Skipped 1 previous similar message [ 6964.197028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6964.209889] Lustre: Skipped 2 previous similar messages [ 6967.264485] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6967.516141] Lustre: server umount lustre-OST0000 complete [ 6974.436714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6974.445600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6974.449412] Lustre: Skipped 1 previous similar message [ 6974.472407] Lustre: Skipped 1 previous similar message [ 6978.663214] Lustre: server umount lustre-MDT0000 complete [ 6979.555082] LustreError: 135987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6979.577971] LustreError: 135987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6983.020879] LustreError: 137958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788429436 with bad export cookie 15793377957870540496 [ 6983.026317] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6983.035027] LustreError: 137958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6983.333715] Lustre: server umount lustre-MDT0001 complete [ 6997.293701] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 7000.485369] Key type lgssc unregistered [ 7000.860844] LNet: 147132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7000.877641] LNetError: 147132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7001.895597] LNet: Removed LNI 192.168.204.116@tcp [ 7003.117494] Key type .llcrypt unregistered [ 7003.119731] Key type ._llcrypt unregistered [ 7024.378588] Key type ._llcrypt registered [ 7024.383578] Key type .llcrypt registered [ 7024.532090] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 7040.385320] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 7041.396211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7041.539905] alg: No test for adler32 (adler32-zlib) [ 7042.686929] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 7043.092377] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 7044.912488] Key type lgssc registered [ 7046.799749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7089.026210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7089.080251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7090.351300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7090.426190] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7090.552441] Lustre: lustre-MDT0000: new disk, initializing [ 7090.633138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7090.661761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7096.290878] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7108.038295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7108.107688] Lustre: 150332:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7108.198736] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7108.202899] Lustre: Skipped 1 previous similar message [ 7108.320274] Lustre: lustre-MDT0001: new disk, initializing [ 7108.480930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7108.527771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7108.554867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7113.136927] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7123.114218] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7129.228412] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7137.078059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7137.398167] Lustre: lustre-OST0000: new disk, initializing [ 7137.405805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7137.412456] Lustre: 152301:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7137.510810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7144.404485] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7147.679992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7147.690311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7147.849517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7154.906418] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7161.533423] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_module obdecho/obdecho [ 7163.317848] LustreError: 153302:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 7163.328750] LustreError: 153302:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 7164.981674] LustreError: 153400:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 7178.723961] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7178.730347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7178.741499] Lustre: Skipped 2 previous similar messages [ 7178.749909] Lustre: Skipped 2 previous similar messages [ 7182.640124] Lustre: server umount lustre-MDT0000 complete [ 7186.196268] LustreError: 150326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788429640 with bad export cookie 5747408519589684873 [ 7186.207443] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7186.217217] LustreError: 150326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7186.721033] Lustre: server umount lustre-MDT0001 complete [ 7201.076907] Lustre: server umount lustre-OST0000 complete [ 7223.150930] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 7233.505139] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 7269.870746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7270.051083] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7270.092424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7270.154831] Lustre: lustre-MDT0000: new disk, initializing [ 7270.230322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7270.249525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7274.137425] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7285.601265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7285.737303] Lustre: 157130:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7285.815579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7285.820650] Lustre: Skipped 1 previous similar message [ 7285.934577] Lustre: lustre-MDT0001: new disk, initializing [ 7286.031873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7286.067163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7286.085719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7291.530394] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7302.065298] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7308.797816] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7316.273502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7316.690606] Lustre: lustre-OST0000: new disk, initializing [ 7316.693606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7316.705418] Lustre: 159096:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7316.832349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7324.168512] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7325.322506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7325.352170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7325.556783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7334.635362] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7339.939424] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7340.252495] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7345.192968] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7345.394822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7351.270560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7351.273260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7351.282442] LustreError: Skipped 1 previous similar message [ 7351.314539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7351.331517] Lustre: Skipped 1 previous similar message [ 7353.694757] Lustre: server umount lustre-OST0000 complete [ 7357.922676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7357.934815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7357.949822] Lustre: Skipped 1 previous similar message [ 7357.960830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7361.505882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7361.506343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7361.523631] Lustre: Skipped 1 previous similar message [ 7363.693636] Lustre: server umount lustre-MDT0000 complete [ 7366.628888] LustreError: 158090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7366.642688] LustreError: 158090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7367.765672] LustreError: 157121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788429821 with bad export cookie 5747408519589685874 [ 7367.776961] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7367.780303] LustreError: 157121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7368.170961] Lustre: server umount lustre-MDT0001 complete [ 7371.982815] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:03:43 (1788429823) [ 7373.670811] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7375.843448] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:03:47 (1788429827) [ 7383.949932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7384.447514] LustreError: 161514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7384.576985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7389.675556] LustreError: 161515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7389.912834] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7394.788279] LustreError: 161514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7398.339128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7403.509449] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7411.836992] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7418.484510] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7426.562229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7426.936661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7426.944359] Lustre: Skipped 1 previous similar message [ 7433.160363] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7440.711917] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7452.551535] Lustre: server umount lustre-MDT0000 complete [ 7453.676849] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7453.695068] LustreError: 164006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7453.698335] Lustre: Skipped 2 previous similar messages [ 7453.732313] LustreError: 164006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7457.196200] LustreError: 163358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788429911 with bad export cookie 5747408519589686798 [ 7457.204647] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7457.225522] LustreError: 163358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7458.796428] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7461.964641] LustreError: 161509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7461.985109] LustreError: 161509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7463.655468] Lustre: server umount lustre-MDT0001 complete [ 7470.249652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7482.996320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7486.603082] LustreError: 165365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7486.613619] LustreError: 165365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7502.817647] Lustre: 165361:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7503.296349] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7508.450234] Lustre: 165363:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7508.462201] Lustre: 165363:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 1 previous similar message [ 7508.489160] Lustre: 161507:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -108 [ 7513.569333] Lustre: 165363:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7516.131410] Lustre: 165361:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7521.250743] LustreError: 165366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7521.250947] Lustre: 165361:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7521.270366] LustreError: 165366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [ 7533.537511] Lustre: 165363:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7533.554949] Lustre: 165363:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 2 previous similar messages [ 7536.160559] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7554.017288] Lustre: 165362:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7554.044408] Lustre: 165362:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 5 previous similar messages [ 7556.192170] Lustre: 161507:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -108 [ 7563.746826] Lustre: 165356:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -108 [ 7584.224252] Lustre: 161507:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.116@tcp: IR log lustre-mdtir failed, not fatal: rc = -108 [ 7588.938850] LustreError: 165366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7588.968639] LustreError: 165366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 46 previous similar messages [ 7589.345884] Lustre: 165362:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 7589.362577] Lustre: 165362:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 8 previous similar messages [ 7604.750753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7609.533563] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7609.832606] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7619.896621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7625.539770] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7625.740387] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7633.550562] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7635.436067] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7642.253295] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7643.887665] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7666.659341] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7666.661328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7666.668062] Lustre: Skipped 1 previous similar message [ 7666.687551] Lustre: server umount lustre-MDT0000 complete [ 7670.487111] LustreError: 163358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788430124 with bad export cookie 5747408519589693539 [ 7670.493270] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7670.503339] LustreError: 163358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7671.195805] Lustre: server umount lustre-MDT0001 complete [ 7686.050204] Lustre: server umount lustre-OST0000 complete [ 7710.152713] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 7721.409644] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 7760.892627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7761.131884] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7761.167984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7761.289860] Lustre: lustre-MDT0000: new disk, initializing [ 7761.432714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7761.437120] Lustre: Skipped 1 previous similar message [ 7761.449489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7766.157591] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7777.191446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7777.305344] Lustre: 170489:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7777.651167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7782.495450] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7792.389849] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7798.702434] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7806.105022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7806.355578] Lustre: lustre-OST0000: new disk, initializing [ 7806.362074] Lustre: Skipped 1 previous similar message [ 7806.368727] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7806.376501] Lustre: Skipped 2 previous similar messages [ 7806.380819] Lustre: 172460:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7812.841626] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7814.763612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7814.772655] Lustre: Skipped 1 previous similar message [ 7814.791930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7814.888076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7822.195331] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7828.349766] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7828.644414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7833.851357] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7834.136390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7840.737782] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7840.745313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7840.756983] LustreError: Skipped 1 previous similar message [ 7840.759958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7840.768959] Lustre: Skipped 3 previous similar messages [ 7840.790763] Lustre: Skipped 3 previous similar messages [ 7842.292097] Lustre: server umount lustre-OST0000 complete [ 7849.444359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7852.459644] Lustre: server umount lustre-MDT0000 complete [ 7856.100835] LustreError: 172103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7856.111608] LustreError: 172103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 28 previous similar messages [ 7856.340887] LustreError: 170483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788430310 with bad export cookie 5747408519589710423 [ 7856.353759] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7856.673145] Lustre: server umount lustre-MDT0001 complete [ 7866.414039] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 06:11:58 (1788430318) [ 7874.015204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7880.528967] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7891.479702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7897.810693] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7905.922444] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7912.566883] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7921.372395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7930.985831] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 7939.906365] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7956.377956] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 06:13:28 (1788430408) [ 7962.086166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7962.101943] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7962.118160] Lustre: Skipped 2 previous similar messages [ 7962.122365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7962.131260] Lustre: Skipped 3 previous similar messages [ 7967.116161] Lustre: server umount lustre-OST0000 complete [ 7977.416823] Lustre: server umount lustre-MDT0000 complete [ 7981.538462] LustreError: 176716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788430435 with bad export cookie 5747408519589711347 [ 7981.539691] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7981.545774] LustreError: 176716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7981.990805] Lustre: server umount lustre-MDT0001 complete [ 7993.689905] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 7996.632342] Key type lgssc unregistered [ 7997.038110] LNet: 178604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7997.043778] LNetError: 178604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7997.054770] LNet: Removed LNI 192.168.204.116@tcp [ 7997.926377] Key type .llcrypt unregistered [ 7997.930351] Key type ._llcrypt unregistered [ 8018.309795] Key type ._llcrypt registered [ 8018.314325] Key type .llcrypt registered [ 8018.439232] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 8019.526078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8019.681844] alg: No test for adler32 (adler32-zlib) [ 8020.789878] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 8021.083748] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 8022.778685] Key type lgssc registered [ 8023.836218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8031.973099] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8031.992083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8033.107165] Lustre: 179857:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 8048.544840] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8053.665688] LustreError: 179844:0:(mgc_request.c:317:do_config_log_add()) mgc192.168.204.116@tcp: failed processing log, type 4: rc = -110 [ 8058.342391] Lustre: 179859:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 8058.848625] LustreError: 179844:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8058.868080] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 8058.879492] LustreError: mgc192.168.204.116@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 8058.897472] LustreError: 179844:0:(tgt_mount.c:2005:server_start_targets()) failed to start server lustre-OST0000: -22 [ 8058.906783] LustreError: 179844:0:(tgt_mount.c:2799:server_fill_super()) Unable to start targets: -22 [ 8058.918056] LustreError: 179844:0:(tgt_mount.c:2249:server_put_super()) no obd lustre-OST0000 [ 8059.038652] Lustre: server umount lustre-OST0000 complete [ 8059.048491] LustreError: 179844:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8068.465543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8068.825888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8073.424996] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8078.315624] LustreError: 180348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8078.340787] LustreError: 180348:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8082.397738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8082.995461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8087.429864] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8095.573683] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8100.970418] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8107.721132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8107.892986] LustreError: 182165:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 8107.905490] Lustre: 182165:0:(mgc_request_server.c:733:mgc_llog_local_copy()) mgc192.168.204.116@tcp: invalid local config log lustre-OST0000: rc = -22 [ 8108.040975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8114.258256] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8122.111539] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8128.993886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8129.009686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8129.038646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8129.514182] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8129.531954] Lustre: Skipped 1 previous similar message [ 8130.968294] Lustre: server umount lustre-MDT0000 complete [ 8134.627985] LustreError: 181076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8134.651262] LustreError: 181076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8134.917888] LustreError: 180331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788430588 with bad export cookie 8803695093550391594 [ 8134.921757] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8134.927220] LustreError: 180331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8135.328975] Lustre: server umount lustre-MDT0001 complete [ 8150.036722] Lustre: server umount lustre-OST0000 complete [ 8161.453572] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 8173.867938] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 8217.216317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8217.463465] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8217.482783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8217.551530] Lustre: lustre-MDT0000: new disk, initializing [ 8217.615306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8217.644413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8223.921830] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8237.524911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8237.686614] Lustre: 185687:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8237.783480] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8237.789373] Lustre: Skipped 1 previous similar message [ 8237.900708] Lustre: lustre-MDT0001: new disk, initializing [ 8238.060704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8238.139909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8238.161926] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8243.254494] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8253.963767] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8260.900982] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8269.898290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8270.260386] Lustre: lustre-OST0000: new disk, initializing [ 8270.271469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8270.285699] Lustre: 187660:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8270.433861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8276.613531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8276.644514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8276.873032] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8277.724178] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8289.654699] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8296.269801] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8296.651356] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8304.559189] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8304.880549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8307.686780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8307.686845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8307.708300] LustreError: Skipped 1 previous similar message [ 8307.714627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8307.714986] Lustre: Skipped 3 previous similar messages [ 8307.812881] Lustre: Skipped 1 previous similar message [ 8313.327424] LustreError: 187964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8313.363519] LustreError: 187964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8313.773837] Lustre: server umount lustre-OST0000 complete [ 8319.970102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8319.979365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8320.000247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8320.004236] Lustre: Skipped 1 previous similar message [ 8323.555992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8323.575773] Lustre: Skipped 1 previous similar message [ 8325.930563] Lustre: server umount lustre-MDT0000 complete [ 8328.684530] LustreError: 187022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8328.719716] LustreError: 187022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8330.392724] LustreError: 186484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788430784 with bad export cookie 8803695093550392371 [ 8330.399105] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8330.411461] LustreError: 186484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8330.870806] Lustre: server umount lustre-MDT0001 complete [ 8343.378829] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:19:54 (1788430794) [ 8366.498286] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 8369.730537] Key type lgssc unregistered [ 8370.093641] LNet: 190607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8370.100502] LNetError: 190607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8370.126955] LNet: Removed LNI 192.168.204.116@tcp [ 8371.243136] Key type .llcrypt unregistered [ 8371.245245] Key type ._llcrypt unregistered [ 8392.522251] Key type ._llcrypt registered [ 8392.524832] Key type .llcrypt registered [ 8392.612841] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 8393.792183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8394.075954] alg: No test for adler32 (adler32-zlib) [ 8395.226528] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 8395.501568] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 8397.225895] Key type lgssc registered [ 8398.556824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8407.150535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8407.179307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8408.716151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8414.766373] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8418.286604] LustreError: 191825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8418.319565] LustreError: 191825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8423.405255] LustreError: 191824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8425.508908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8426.250172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8432.241340] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8442.111134] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8450.792410] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8461.144523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8461.517319] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8461.528564] LustreError: 193690:0:(llog.c:1696:llog_backup()) mgc192.168.204.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8461.543399] Lustre: 193690:0:(mgc_request_server.c:730:mgc_llog_local_copy()) mgc192.168.204.116@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8461.689585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8471.371213] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8481.154600] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8484.326056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8484.337154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8484.346202] Lustre: Skipped 2 previous similar messages [ 8484.349018] Lustre: Skipped 1 previous similar message [ 8489.449724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8489.463835] Lustre: Skipped 3 previous similar messages [ 8490.598139] Lustre: server umount lustre-MDT0000 complete [ 8494.566844] LustreError: 191824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8494.586833] LustreError: 191824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8495.365991] LustreError: 191804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788430949 with bad export cookie 2874132144957685814 [ 8495.366344] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8495.389580] LustreError: 191804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8495.769760] Lustre: server umount lustre-MDT0001 complete [ 8511.326903] Lustre: server umount lustre-OST0000 complete [ 8523.692259] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 8535.538601] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 8581.484181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8581.990296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8582.061187] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8582.195351] Lustre: lustre-MDT0000: new disk, initializing [ 8582.286140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8582.309950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8587.607305] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8601.297474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8601.457271] Lustre: 197218:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8601.647669] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8601.651048] Lustre: Skipped 1 previous similar message [ 8601.663492] LustreError: 198001:0:(llog.c:1696:llog_backup()) mgc192.168.204.116@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8601.669610] LustreError: 198001:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8601.676060] Lustre: 198001:0:(mgc_request_server.c:753:mgc_llog_local_copy()) mgc192.168.204.116@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8601.723670] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8601.734451] Lustre: Skipped 1 previous similar message [ 8601.879534] Lustre: lustre-MDT0001: new disk, initializing [ 8602.000155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8602.051811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8602.081695] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8607.380115] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8618.102343] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8625.094668] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8633.258563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8633.494133] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8633.496215] Lustre: Skipped 1 previous similar message [ 8633.501916] LustreError: 199163:0:(llog.c:1696:llog_backup()) mgc192.168.204.116@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8633.510193] LustreError: 199163:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8633.514986] Lustre: 199163:0:(mgc_request_server.c:753:mgc_llog_local_copy()) mgc192.168.204.116@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8633.521376] Lustre: 199163:0:(mgc_request_server.c:753:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8633.540279] Lustre: lustre-OST0000: new disk, initializing [ 8633.544863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8633.552234] Lustre: 199187:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8633.660766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8641.123112] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8641.522810] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8641.526841] Lustre: Skipped 1 previous similar message [ 8641.530424] LustreError: 197222:0:(llog.c:1696:llog_backup()) mgc192.168.204.116@tcp: failed to backup log lustre-client: rc = -28 [ 8641.539959] LustreError: 197222:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8641.544370] Lustre: 197222:0:(mgc_request_server.c:753:mgc_llog_local_copy()) mgc192.168.204.116@tcp: failed to copy new config lustre-client: rc = -28 [ 8641.555139] Lustre: 197222:0:(mgc_request_server.c:753:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8641.652194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8641.663198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8641.825503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8652.669613] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8660.344039] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8660.785899] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8668.144955] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8668.568863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8672.739641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8672.748210] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8672.789271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8672.821128] Lustre: Skipped 1 previous similar message [ 8677.866791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8677.890016] Lustre: Skipped 2 previous similar messages [ 8678.499419] Lustre: server umount lustre-OST0000 complete [ 8684.002077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8684.008478] LustreError: Skipped 1 previous similar message [ 8684.011326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8684.018312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8688.100122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8689.577510] Lustre: server umount lustre-MDT0000 complete [ 8693.229202] LustreError: 197225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8693.267772] LustreError: 197225:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8694.188162] LustreError: 199196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788431148 with bad export cookie 2874132144957686591 [ 8694.190243] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8694.217741] LustreError: 199196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8694.609799] Lustre: server umount lustre-MDT0001 complete [ 8706.015290] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 06:25:57 (1788431157) [ 8728.008571] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 8742.508049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8743.255876] LustreError: 202321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8743.335956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8749.925025] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8753.640989] LustreError: 202322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8753.679737] LustreError: 202322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8763.163519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8769.390963] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8772.797113] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8781.914137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8782.604524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8782.615154] Lustre: Skipped 1 previous similar message [ 8792.034559] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8803.556920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8803.712986] Lustre: lustre-OST0001: new disk, initializing [ 8803.720223] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8803.746865] Lustre: 204631:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8812.055219] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8813.524241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8813.552424] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8813.836274] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8827.002551] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8836.408619] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8839.026584] Lustre: DEBUG MARKER: ADD OST3 [ 8845.999112] loop: module loaded [ 8846.148445] loop0: detected capacity change from 0 to 204800000 [ 8846.561655] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8846.570633] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8846.582646] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8854.686309] Lustre: DEBUG MARKER: STOP OST3 [ 8855.054780] loop0: detected capacity change from 0 to 204800000 [ 8862.758141] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8863.141851] Lustre: lustre-OST0003: new disk, initializing [ 8863.147701] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8863.162574] Lustre: 206711:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8863.396985] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8863.409972] Lustre: Skipped 1 previous similar message [ 8864.736355] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8864.958112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8864.967467] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8865.015577] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8866.369548] Lustre: Failing over lustre-OST0003 [ 8866.790629] LustreError: 206800:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8866.884762] Lustre: server umount lustre-OST0003 complete [ 8870.053728] LustreError: 204052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8870.094685] LustreError: 204052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8870.391188] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8870.396884] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8872.290312] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8886.321973] Lustre: DEBUG MARKER: START OST3 again [ 8896.662467] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8897.343927] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8899.101118] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8899.208489] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8899.208662] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8907.871237] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 8933.650353] Lustre: Failing over lustre-OST0003 [ 8933.806663] Lustre: server umount lustre-OST0003 complete [ 8935.394375] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8935.403176] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8935.427788] LustreError: 203806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8935.435477] Lustre: Skipped 2 previous similar messages [ 8935.497281] LustreError: 203806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 8953.930804] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8953.973160] Lustre: Skipped 1 previous similar message [ 8953.978860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8953.999101] Lustre: Skipped 4 previous similar messages [ 8958.435254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8958.458894] LustreError: Skipped 1 previous similar message [ 8960.384940] Lustre: server umount lustre-MDT0000 complete [ 8965.990518] LustreError: 202301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788431419 with bad export cookie 2874132144957687515 [ 8965.994402] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8966.012468] LustreError: 202301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8970.237838] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8970.246623] Lustre: Skipped 7 previous similar messages [ 8972.628948] Lustre: server umount lustre-MDT0001 complete [ 8978.142217] Lustre: server umount lustre-OST0000 complete [ 8984.615777] Lustre: server umount lustre-OST0001 complete [ 8994.541602] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 9008.697789] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 9063.516869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9063.953408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9063.995526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9064.111063] Lustre: lustre-MDT0000: new disk, initializing [ 9064.336656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9064.345356] Lustre: Skipped 1 previous similar message [ 9064.394528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9071.264694] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9084.683168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9084.823856] Lustre: 212254:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9084.927527] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9084.935698] Lustre: Skipped 1 previous similar message [ 9084.991178] Lustre: lustre-MDT0001: new disk, initializing [ 9085.145688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9085.168620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9091.107274] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9103.255350] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9110.660396] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9120.026990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9120.525813] Lustre: lustre-OST0000: new disk, initializing [ 9120.534955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9120.553318] Lustre: 214223:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9122.581086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9122.606355] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9122.849159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9129.652904] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9141.172748] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9146.872229] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9147.357654] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9154.789396] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9155.016855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9158.641762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9158.671768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9158.685950] LustreError: Skipped 1 previous similar message [ 9158.721730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9158.730765] Lustre: Skipped 4 previous similar messages [ 9164.862777] Lustre: server umount lustre-OST0000 complete [ 9177.938374] Lustre: server umount lustre-MDT0000 complete [ 9179.108403] LustreError: 212265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9179.138187] LustreError: 212265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [ 9182.735244] LustreError: 213047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788431636 with bad export cookie 2874132144957690147 [ 9182.776423] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9183.493444] Lustre: server umount lustre-MDT0001 complete [ 9187.633642] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 06:33:59 (1788431639) [ 9207.296662] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 9219.838297] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 9263.712038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9264.029772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9264.101082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9264.257981] Lustre: lustre-MDT0000: new disk, initializing [ 9264.427947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9270.902373] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9284.625056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9284.756355] Lustre: 218837:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9285.246287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9290.893936] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9302.756328] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9309.709127] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9318.528783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9318.853864] Lustre: 220808:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9320.409690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9320.547386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9325.791578] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9336.910824] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9343.422956] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9343.755358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9350.574867] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9350.883898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9356.264176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9356.266096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9356.273830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9356.273840] Lustre: Skipped 6 previous similar messages [ 9356.289516] Lustre: Skipped 3 previous similar messages [ 9356.332784] LustreError: Skipped 1 previous similar message [ 9368.036366] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9368.173949] Lustre: server umount lustre-OST0000 complete [ 9379.333762] Lustre: server umount lustre-MDT0000 complete [ 9381.869777] LustreError: 221144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9381.901510] LustreError: 221144:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9384.667226] LustreError: 219634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788431838 with bad export cookie 2874132144957691064 [ 9384.680055] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9384.687448] LustreError: 219634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9385.396933] Lustre: server umount lustre-MDT0001 complete [ 9397.086591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9397.950356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9397.960434] Lustre: Skipped 5 previous similar messages [ 9403.450478] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9414.342803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9420.189387] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9431.121499] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9438.847608] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9448.918308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9458.658370] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9469.200943] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9480.676915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9480.685932] LustreError: Skipped 1 previous similar message [ 9486.779403] Lustre: server umount lustre-OST0000 complete [ 9491.433110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9491.463160] Lustre: Skipped 4 previous similar messages [ 9491.471446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9491.484525] Lustre: Skipped 10 previous similar messages [ 9497.526637] Lustre: server umount lustre-MDT0000 complete [ 9502.631704] LustreError: 223012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788431956 with bad export cookie 2874132144957691995 [ 9502.644258] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9502.649962] LustreError: 223012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9503.211471] Lustre: server umount lustre-MDT0001 complete [ 9611.958518] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [ 9615.322433] Key type lgssc unregistered [ 9615.623827] LNet: 226697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9615.636707] LNetError: 226697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9615.660817] LNet: Removed LNI 192.168.204.116@tcp [ 9616.771534] Key type .llcrypt unregistered [ 9616.774271] Key type ._llcrypt unregistered [ 9647.381153] Key type ._llcrypt registered [ 9647.388915] Key type .llcrypt registered [ 9647.552960] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [ 9666.699952] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 9668.587396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9668.673433] alg: No test for adler32 (adler32-zlib) [ 9669.750326] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 9670.021234] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [ 9671.809729] Key type lgssc registered [ 9673.266653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9716.096507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9716.153722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9717.396509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9717.432584] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9717.540150] Lustre: lustre-MDT0000: new disk, initializing [ 9717.629890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9717.647808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9722.755783] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9735.543864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9735.641060] Lustre: 230052:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9735.733422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9735.736486] Lustre: Skipped 1 previous similar message [ 9735.833519] Lustre: lustre-MDT0001: new disk, initializing [ 9735.946985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9735.996864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9736.014985] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9741.723791] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9753.583365] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9760.998419] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9769.112376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9769.500461] Lustre: lustre-OST0000: new disk, initializing [ 9769.506078] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9769.523435] Lustre: 232020:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9769.654479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9777.246316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9777.263689] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9777.399808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9778.901366] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9789.842818] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9795.376494] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9795.645692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9802.330098] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9802.592091] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9808.359749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9808.362132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9808.375228] LustreError: Skipped 1 previous similar message [ 9808.376502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9808.401140] Lustre: Skipped 1 previous similar message [ 9811.347362] Lustre: server umount lustre-OST0000 complete [ 9818.086478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9818.095224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9818.131183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9818.138434] Lustre: Skipped 1 previous similar message [ 9822.173937] Lustre: server umount lustre-MDT0000 complete [ 9826.536978] LustreError: 230045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788432280 with bad export cookie 9137959524008421693 [ 9826.539760] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9826.548942] LustreError: 230045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9826.863275] Lustre: server umount lustre-MDT0001 complete [ 9848.278384] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [ 9859.457717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9859.837734] LustreError: 234959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9859.853914] LustreError: 234959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9859.888657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9864.576393] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9865.192362] LustreError: 234960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9869.351404] LustreError: 234959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9874.408897] LustreError: 234960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9874.596920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9875.039255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9881.365457] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9884.572208] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9894.272696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9894.959604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9902.940388] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9913.316996] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9913.783626] Lustre: lustre-OST0001: new disk, initializing [ 9913.802704] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9913.810834] Lustre: 237267:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9913.933080] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9919.732430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9919.750031] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9919.845624] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9920.924909] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [ 9933.301976] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9942.871742] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9959.079152] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 06:46:50 (1788432410) [ 9961.955985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9961.971636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9961.999645] Lustre: Skipped 1 previous similar message [ 9962.006639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9962.012820] Lustre: Skipped 1 previous similar message [ 9966.056597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9966.061992] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9966.088970] Lustre: Skipped 2 previous similar messages [ 9971.172768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9971.180390] Lustre: Skipped 3 previous similar messages [ 9976.298199] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9976.811208] Lustre: server umount lustre-MDT0000 complete [ 9980.932274] LustreError: 234939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788432434 with bad export cookie 9137959524008422617 [ 9980.938667] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9980.949096] LustreError: 234939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9981.374535] Lustre: server umount lustre-MDT0001 complete [ 9995.600125] Lustre: server umount lustre-OST0000 complete [ 9997.796539] Lustre: 228270:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788432435/real 1788432435] req@ffff8b5e093cea00 x1875307001935744/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788432451 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9997.825247] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9999.628964] Lustre: server umount lustre-OST0001 complete [10007.561681] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [10019.700374] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [10068.404134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10068.671551] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10068.701314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10068.813434] Lustre: lustre-MDT0000: new disk, initializing [10068.927301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10068.956543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10074.921241] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10089.066127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10089.197506] Lustre: 241218:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10089.327810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10089.335392] Lustre: Skipped 1 previous similar message [10089.495727] Lustre: lustre-MDT0001: new disk, initializing [10089.703467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10089.719839] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10095.347090] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10106.993965] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10114.300634] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10122.192806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10122.440574] Lustre: lustre-OST0000: new disk, initializing [10122.446467] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10122.454707] Lustre: 243190:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10123.972571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10123.997562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10124.237229] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10130.134450] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10140.787948] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10146.216332] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10146.457624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10151.632956] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10151.899870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10154.983818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10154.999142] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10155.023976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10155.034630] Lustre: Skipped 5 previous similar messages [10160.861330] Lustre: server umount lustre-OST0000 complete [10166.764257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10166.781465] LustreError: Skipped 1 previous similar message [10171.932546] Lustre: server umount lustre-MDT0000 complete [10175.463363] LustreError: 241231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10175.497455] LustreError: 241231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10177.200833] LustreError: 243189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788432631 with bad export cookie 9137959524008424927 [10177.216035] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10177.218819] LustreError: 243189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10177.759703] Lustre: server umount lustre-MDT0001 complete [10189.069865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10189.558452] LustreError: 245408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10189.645631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10189.651507] Lustre: Skipped 2 previous similar messages [10195.409548] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10205.447078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10210.282930] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10218.336170] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10224.970729] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10232.167694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10240.784358] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10250.380536] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10398.689328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10398.694444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10398.709881] Lustre: Skipped 3 previous similar messages [10398.722601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10398.738076] Lustre: Skipped 5 previous similar messages [10402.200353] Lustre: server umount lustre-OST0000 complete [10421.218634] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10421.390878] Lustre: server umount lustre-MDT0000 complete [10424.290353] LustreError: 247749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10424.308660] LustreError: 247749:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10425.584493] LustreError: 245388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788432879 with bad export cookie 9137959524008425851 [10425.593832] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10425.603864] LustreError: 245388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10425.982354] Lustre: server umount lustre-MDT0001 complete [10520.607932] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [10523.845634] Key type lgssc unregistered [10524.133804] LNet: 249016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10524.164324] LNetError: 249016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10525.232854] LNet: Removed LNI 192.168.204.116@tcp [10526.381386] Key type .llcrypt unregistered [10526.385596] Key type ._llcrypt unregistered [10557.343910] Key type ._llcrypt registered [10557.348808] Key type .llcrypt registered [10557.554747] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [10576.237565] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [10577.074583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10577.138752] alg: No test for adler32 (adler32-zlib) [10578.342697] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [10578.585139] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [10580.258270] Key type lgssc registered [10581.561059] Lustre: Echo OBD driver; http://www.lustre.org/ [10640.754203] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [10653.413134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10653.437696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10654.683449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10654.713314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10654.776384] Lustre: lustre-MDT0000: new disk, initializing [10654.886578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10654.940807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10659.689992] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10673.264060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10673.472592] Lustre: 253111:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10673.644272] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10673.647862] Lustre: Skipped 1 previous similar message [10673.760170] Lustre: lustre-MDT0001: new disk, initializing [10673.912891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10673.994923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10674.017671] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10679.704962] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10686.587054] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10694.465815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10695.065601] Lustre: lustre-OST0000: new disk, initializing [10695.091388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10695.113879] Lustre: 254743:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10695.277657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10697.245219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10697.267418] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10697.415510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10702.351641] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10715.921535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10716.174559] Lustre: lustre-OST0001: new disk, initializing [10716.178385] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10716.185687] Lustre: 255616:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10716.340965] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10724.239147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10724.264880] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10724.466465] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10724.648969] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10736.890499] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10742.848424] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10760.166248] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10760.184907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10760.185053] Lustre: Skipped 3 previous similar messages [10760.195801] Lustre: Skipped 3 previous similar messages [10762.574301] Lustre: server umount lustre-MDT0000 complete [10765.286377] LustreError: 253121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10765.307256] LustreError: 253121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10766.691201] LustreError: 253102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788433220 with bad export cookie 7488435153061087580 [10766.703345] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10766.711170] LustreError: 253102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10767.109387] Lustre: server umount lustre-MDT0001 complete [10781.681598] Lustre: server umount lustre-OST0000 complete [10796.090352] Lustre: server umount lustre-OST0001 complete [10810.548561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10822.060982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10832.868285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10845.341901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10869.814521] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [10880.541566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10880.578504] Lustre: lustre-MDT0000: reset Object Index mappings [10881.140083] LustreError: 260642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10881.244326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10885.933484] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10886.630938] LustreError: 260643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10891.753915] LustreError: 260642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10896.168981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10896.183433] Lustre: lustre-MDT0001: reset Object Index mappings [10896.646779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10902.056727] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10905.346457] Lustre: 261787:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10912.629521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10913.186673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10919.804101] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10921.400331] LustreError: 262143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10926.592284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10929.544313] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10935.309744] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10936.000440] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [10945.569307] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10949.843177] Lustre: 263660:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10958.307321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10958.322192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10958.351278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10960.874197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10960.874378] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10960.896741] Lustre: Skipped 2 previous similar messages [10962.288961] Lustre: server umount lustre-MDT0000 complete [10965.996015] LustreError: 261631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10966.019016] LustreError: 261631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10966.635764] LustreError: 260624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788433420 with bad export cookie 7488435153061094531 [10966.639133] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10966.651346] LustreError: 260624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10967.160867] Lustre: server umount lustre-MDT0001 complete [10981.395033] Lustre: server umount lustre-OST0000 complete [10996.425333] Lustre: server umount lustre-OST0001 complete [11018.612817] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 07:04:30 (1788433470) [11042.228985] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [11055.749324] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [11098.007333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11098.307243] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11098.340678] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11098.416670] Lustre: lustre-MDT0000: new disk, initializing [11098.499086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11098.503347] Lustre: Skipped 1 previous similar message [11098.521229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11103.538748] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11115.237730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11115.331825] Lustre: 267920:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11115.398994] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11115.402016] Lustre: Skipped 1 previous similar message [11115.465595] Lustre: lustre-MDT0001: new disk, initializing [11115.592664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11115.605174] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11119.991800] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11130.273058] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11136.665402] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11143.623456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11143.894120] Lustre: lustre-OST0000: new disk, initializing [11143.903530] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11143.912070] Lustre: 269887:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11145.025164] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11145.037539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11145.189351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11150.941565] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11161.683428] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11167.708568] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11168.087632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11174.524790] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11174.949067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11181.028516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11181.029253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11181.032125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11181.032132] Lustre: Skipped 3 previous similar messages [11181.041453] Lustre: Skipped 1 previous similar message [11183.419937] Lustre: server umount lustre-OST0000 complete [11191.271898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11191.272876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11191.281602] Lustre: Skipped 1 previous similar message [11191.291753] Lustre: Skipped 2 previous similar messages [11193.850953] Lustre: server umount lustre-MDT0000 complete [11196.389245] LustreError: 267930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11196.402114] LustreError: 267930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11197.606941] LustreError: 267910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788433651 with bad export cookie 7488435153061096736 [11197.610122] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11197.619203] LustreError: 267910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11198.000471] Lustre: server umount lustre-MDT0001 complete [11219.599952] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [11231.093840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11231.587253] LustreError: 272824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11231.672274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11231.680547] Lustre: Skipped 2 previous similar messages [11236.469660] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11246.681384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11252.646940] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11256.432986] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [11264.773319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11272.804946] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11282.717417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11282.877284] Lustre: lustre-OST0001: new disk, initializing [11282.881291] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11282.887685] Lustre: 275137:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11285.098113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11285.118542] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11285.245582] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11290.495098] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11301.752419] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11306.477969] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11313.509307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [11313.523504] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [11328.993693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11329.005492] LustreError: Skipped 1 previous similar message [11329.022709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11329.043468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11333.379447] Lustre: server umount lustre-MDT0000 complete [11336.682265] LustreError: 272821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11336.721192] LustreError: 272821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11338.235391] LustreError: 272804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788433792 with bad export cookie 7488435153061097660 [11338.241767] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11338.252548] LustreError: 272804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11338.724588] Lustre: server umount lustre-MDT0001 complete [11353.654099] Lustre: server umount lustre-OST0000 complete [11357.664262] Lustre: 250595:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788433795/real 1788433795] req@ffff8b5d019aca80 x1875307955234176/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788433811 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11357.693239] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11357.706558] Lustre: Skipped 3 previous similar messages [11358.205285] Lustre: server umount lustre-OST0001 complete [11378.583068] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [11390.568249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11391.214445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11391.221924] Lustre: Skipped 3 previous similar messages [11396.672493] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11405.952364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11411.740353] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11414.934899] Lustre: 279626:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11422.482846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11430.719544] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11436.541768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11441.307530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11447.319586] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11451.539225] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11460.381503] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11464.856453] Lustre: 281502:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11471.192957] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11471.279481] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11486.504574] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 07:12:18 (1788433938) [11503.589570] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11503.592721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11503.593394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11503.626988] Lustre: Skipped 3 previous similar messages [11503.655106] Lustre: Skipped 7 previous similar messages [11508.737803] LustreError: 282391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11508.756861] LustreError: 282391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [11508.899991] Lustre: server umount lustre-MDT0000 complete [11513.802347] LustreError: 278465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788433967 with bad export cookie 7488435153061104632 [11513.811802] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11520.446654] Lustre: server umount lustre-MDT0001 complete [11531.704479] Lustre: server umount lustre-OST0000 complete [11536.490303] Lustre: server umount lustre-OST0001 complete [11545.949652] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [11561.170973] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [11620.026788] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [11630.796571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11630.970292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11631.008808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11631.088541] Lustre: lustre-MDT0000: new disk, initializing [11631.186023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11636.117993] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11648.075787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11648.249491] Lustre: 286079:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11648.488234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11648.492607] Lustre: Skipped 4 previous similar messages [11648.525740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11653.905722] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11666.564265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11666.851201] Lustre: 287713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11668.163498] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11674.054205] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11686.664826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11686.842881] Lustre: 288582:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11688.296331] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11688.321094] Lustre: Skipped 1 previous similar message [11688.538057] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11695.461915] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11706.673548] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11711.152505] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11711.161720] Lustre: Skipped 1 previous similar message [11728.068265] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 07:16:19 (1788434179) [11730.401547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11730.410513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11730.418755] Lustre: Skipped 2 previous similar messages [11730.427663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11730.432027] Lustre: Skipped 4 previous similar messages [11732.680986] Lustre: server umount lustre-MDT0000 complete [11736.502257] LustreError: 286070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788434190 with bad export cookie 7488435153061112234 [11736.503173] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11736.512079] LustreError: 286070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11736.802741] Lustre: server umount lustre-MDT0001 complete [11751.023530] Lustre: server umount lustre-OST0000 complete [11766.293400] Lustre: server umount lustre-OST0001 complete [11776.737975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11777.154870] LustreError: 290833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11777.174985] LustreError: 290833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11782.508791] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11793.030356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11798.633941] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11807.478818] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11813.882792] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11821.888450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11829.324695] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11838.004921] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11858.169664] loop1: detected capacity change from 0 to 204800000 [11858.484494] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11858.500928] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11858.527446] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11863.767688] loop1: detected capacity change from 0 to 204800000 [11869.694636] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11869.749248] LustreError: 290822:0:(mgs_llog.c:680:mgs_find_or_make_fsdb_nolock()) fs15853: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11869.761382] LustreError: 290822:0:(mgs_llog.c:739:mgs_check_index()) Can't get db for fs15853 [11869.772912] LustreError: 290822:0:(mgs_handler.c:517:mgs_target_reg()) fs15853-MDT0001: Error checking target index: rc = -13 [11869.789311] LustreError: fs15853-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11869.803622] LustreError: 294551:0:(tgt_mount.c:2799:server_fill_super()) Unable to start targets: -13 [11869.812931] LustreError: 294551:0:(tgt_mount.c:2249:server_put_super()) no obd fs15853-MDT0001 [11869.817250] LustreError: 294551:0:(tgt_mount.c:130:server_deregister_mount()) fs15853-MDT0001 not registered [11869.844551] Lustre: server umount fs15853-MDT0001 complete [11869.848652] LustreError: 294551:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11879.069574] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11879.214381] Lustre: Setting parameter fs15853-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15853-MDT0001 [11879.306421] Lustre: srv-fs15853-MDT0001: No data found on store. Initialize space. [11879.312678] Lustre: Skipped 4 previous similar messages [11879.397294] Lustre: fs15853-MDT0001: new disk, initializing [11879.401074] Lustre: Skipped 3 previous similar messages [11879.552632] Lustre: srv-fs15853-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11879.567336] Lustre: 295019:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15853-MDT0001-mdtlov: can't create new llog: rc = -115 [11879.584457] LustreError: 295019:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15853-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11885.182935] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11892.676956] Lustre: server umount fs15853-MDT0001 complete [11905.532791] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11909.167857] loop1: detected capacity change from 0 to 204800000 [11917.039077] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11917.868792] Lustre: 296589:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15853-MDT0001-mdtlov: can't create new llog: rc = -115 [11917.903476] LustreError: 296589:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15853-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11923.374655] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11928.565975] Lustre: server umount fs15853-MDT0001 complete [11941.857516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11941.865045] LustreError: Skipped 1 previous similar message [11941.869340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11941.894068] Lustre: Skipped 3 previous similar messages [11941.905626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11947.781725] Lustre: server umount lustre-MDT0000 complete [11957.726957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11957.868621] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11963.132955] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11963.385404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11971.997304] loop1: detected capacity change from 0 to 204800000 [11977.956237] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11978.346876] Lustre: 299037:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs15853-MDT0001-mdtlov: can't create new llog: rc = -115 [11978.354053] LustreError: 299037:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs15853-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11983.337899] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [11989.246699] Lustre: server umount fs15853-MDT0001 complete [12019.707341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12026.017440] Lustre: server umount lustre-OST0000 complete [12037.001384] Lustre: server umount lustre-MDT0000 complete [12041.525312] LustreError: 290815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788434495 with bad export cookie 7488435153061115097 [12041.529911] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12041.538782] LustreError: 290815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12042.197204] Lustre: server umount lustre-MDT0001 complete [12055.935879] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [12058.753915] Key type lgssc unregistered [12059.106363] LNet: 301382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12059.117730] LNetError: 301382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12059.132661] LNet: Removed LNI 192.168.204.116@tcp [12060.040159] Key type .llcrypt unregistered [12060.043629] Key type ._llcrypt unregistered [12083.289179] Key type ._llcrypt registered [12083.296638] Key type .llcrypt registered [12083.387377] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [12084.472423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12084.516174] alg: No test for adler32 (adler32-zlib) [12085.676760] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [12085.956397] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [12087.673198] Key type lgssc registered [12089.019634] Lustre: Echo OBD driver; http://www.lustre.org/ [12097.069179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12097.084773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12098.675885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12103.170439] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12108.259337] LustreError: 302651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12108.284918] LustreError: 302651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12112.610367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12113.103731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12117.961585] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12125.989333] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12131.534429] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12139.955694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12140.342064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12143.404982] LustreError: 304480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12146.889585] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12148.726996] LustreError: 304480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12153.830554] LustreError: 304662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12154.970539] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12157.876675] LustreError: 304480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12168.165066] LustreError: 304481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12168.181505] LustreError: 304481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [12178.645168] loop1: detected capacity change from 0 to 204800000 [12178.876834] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12178.894518] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12178.909948] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12184.232870] loop1: detected capacity change from 0 to 204800000 [12188.647094] LustreError: 304481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12188.659524] LustreError: 304481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [12189.660638] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12189.871840] Lustre: fs17202-OST0000: new disk, initializing [12189.875914] Lustre: srv-fs17202-OST0000: No data found on store. Initialize space. [12189.887245] Lustre: 306388:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12189.998480] Lustre: fs17202-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12196.146442] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12204.042915] Lustre: server umount fs17202-OST0000 complete [12216.498846] loop1: detected capacity change from 0 to 204800000 [12221.790314] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12221.992534] Lustre: fs17202-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12224.487110] LustreError: 304662:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12224.512575] LustreError: 304662:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [12224.528744] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12228.987818] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12239.001641] loop2: detected capacity change from 0 to 204800000 [12239.341867] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12239.348964] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12239.365581] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12239.873291] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [12246.149183] loop2: detected capacity change from 0 to 204800000 [12252.700830] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12252.781519] LustreError: fs17202-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [12252.788699] LustreError: fs17202-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [12252.805725] LustreError: 309432:0:(tgt_mount.c:2799:server_fill_super()) Unable to start targets: -13 [12252.816440] LustreError: 309432:0:(tgt_mount.c:2249:server_put_super()) no obd fs17202-OST0001 [12252.822393] LustreError: 309432:0:(tgt_mount.c:130:server_deregister_mount()) fs17202-OST0001 not registered [12252.850823] Lustre: server umount fs17202-OST0001 complete [12252.856056] LustreError: 309432:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [12264.894029] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12265.172199] Lustre: fs17202-OST0001: new disk, initializing [12265.180541] Lustre: srv-fs17202-OST0001: No data found on store. Initialize space. [12265.203849] Lustre: 309984:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12265.294824] Lustre: fs17202-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12273.389656] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12284.211339] Lustre: server umount fs17202-OST0000 complete [12290.142682] LustreError: 304480:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12290.157747] LustreError: 304480:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [12296.047916] Lustre: server umount fs17202-OST0001 complete [12308.399239] loop1: detected capacity change from 0 to 204800000 [12314.402488] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12314.625370] Lustre: fs17202-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12322.461532] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12329.566333] loop2: detected capacity change from 0 to 204800000 [12336.954612] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12346.117581] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12358.008541] Lustre: server umount fs17202-OST0000 complete [12368.952548] Lustre: server umount fs17202-OST0001 complete [12378.093773] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [12384.230207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12384.244075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12384.276886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12388.326712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12388.327757] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12388.346507] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12389.787708] Lustre: server umount lustre-OST0000 complete [12394.470599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12394.491143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12394.495550] Lustre: Skipped 1 previous similar message [12398.564498] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12398.565488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12398.615971] Lustre: Skipped 1 previous similar message [12399.930183] Lustre: server umount lustre-MDT0000 complete [12404.215565] LustreError: 302633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788434858 with bad export cookie 10726410750204950191 [12404.225411] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12404.237132] LustreError: 302633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12404.703689] Lustre: server umount lustre-MDT0001 complete [12417.306519] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [12420.866699] Key type lgssc unregistered [12421.203671] LNet: 315909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12421.210642] LNetError: 315909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12421.228939] LNet: Removed LNI 192.168.204.116@tcp [12422.140158] Key type .llcrypt unregistered [12422.143324] Key type ._llcrypt unregistered [12451.129403] Key type ._llcrypt registered [12451.131922] Key type .llcrypt registered [12451.253616] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [12452.630287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12452.667047] alg: No test for adler32 (adler32-zlib) [12453.702462] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [12453.935561] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [12455.608129] Key type lgssc registered [12456.631102] Lustre: Echo OBD driver; http://www.lustre.org/ [12465.877206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12465.895322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12467.433738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12471.998129] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12477.931044] LustreError: 317492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12477.957177] LustreError: 317492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12481.914859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12482.619482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12487.631456] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12490.564486] Lustre: 318636:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [12498.245773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12498.875892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12505.774627] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12508.071108] LustreError: 318987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12512.163166] LustreError: 318988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12512.180171] LustreError: 318988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12514.838138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12515.281048] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12521.793662] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12531.945907] Lustre: DEBUG MARKER: Using TIMEOUT=20 [12542.829776] Lustre: 320572:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [12549.354340] Lustre: DEBUG MARKER: === conf-sanity: finish setup 07:30:01 (1788435001) === [12563.429745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12563.441478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12563.464734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12564.450670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12564.460118] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12564.486908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12565.697550] Lustre: server umount lustre-OST0000 complete [12566.506084] LustreError: 320275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12566.542342] LustreError: 320275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12571.622094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12571.623822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12571.624025] LustreError: 318989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12571.637184] Lustre: Skipped 1 previous similar message [12571.659329] Lustre: Skipped 2 previous similar messages [12576.135350] Lustre: server umount lustre-MDT0000 complete [12579.779944] LustreError: 317473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788435033 with bad export cookie 17178370171187410496 [12579.787308] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12579.794137] LustreError: 317473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12580.240814] Lustre: server umount lustre-MDT0001 complete [12598.048153] Lustre: 316755:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788435035/real 1788435035] req@ffff8b5e074f5f80 x1875309921118464/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788435051 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12598.075105] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12598.086266] Lustre: Skipped 1 previous similar message [12600.292354] Lustre: 316757:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788435038/real 1788435038] req@ffff8b5d03134380 x1875309921118592/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788435054 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12602.400220] Lustre: 316757:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788435040/real 1788435040] req@ffff8b5e1f4ab480 x1875309921118720/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788435056 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12605.414349] Lustre: 316756:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788435043/real 1788435043] req@ffff8b5e1f4a8700 x1875309921118976/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788435059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12611.616252] Lustre: 316756:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788435049/real 1788435049] req@ffff8b5d03135500 x1875309921119104/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788435065 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [12686.580210] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [12762.418110] Lustre: server umount lustre-OST0001 complete [12769.455486] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [12783.402708] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [12816.566873] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 07:34:28 (1788435268) [12832.575070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12832.846662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12832.873098] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12832.985613] Lustre: lustre-MDT0000: new disk, initializing [12833.120587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12833.146822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12837.498133] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12848.534569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12848.657736] Lustre: 325489:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12848.805154] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12848.809540] Lustre: Skipped 1 previous similar message [12848.906601] Lustre: lustre-MDT0001: new disk, initializing [12849.066812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12849.143541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12849.162990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12853.815314] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12864.525419] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12870.562818] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12877.899018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12878.418603] Lustre: lustre-OST0000: new disk, initializing [12878.424604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12878.436166] Lustre: 327460:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12878.562785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12879.889141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12879.896551] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12880.024490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12886.745096] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12898.340463] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12915.684751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12915.689186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12915.698664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12920.750760] Lustre: server umount lustre-OST0000 complete [12925.921301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12925.934486] LustreError: Skipped 1 previous similar message [12925.939053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12925.953328] Lustre: Skipped 1 previous similar message [12925.959881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12925.970258] Lustre: Skipped 2 previous similar messages [12930.830381] Lustre: server umount lustre-MDT0000 complete [12931.562574] LustreError: 325502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12931.583634] LustreError: 325502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12934.866420] LustreError: 326296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788435388 with bad export cookie 17178370171187411973 [12934.871835] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12934.880053] LustreError: 326296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12935.325821] Lustre: server umount lustre-MDT0001 complete [12945.340931] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [12947.790300] Key type lgssc unregistered [12948.058895] LNet: 329365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12948.064466] LNetError: 329365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12948.097921] LNet: Removed LNI 192.168.204.116@tcp [12948.833347] Key type .llcrypt unregistered [12948.842286] Key type ._llcrypt unregistered [12969.813828] Key type ._llcrypt registered [12969.816604] Key type .llcrypt registered [12969.921376] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [12971.031487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12971.188109] alg: No test for adler32 (adler32-zlib) [12972.319793] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [12972.556867] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [12974.232176] Key type lgssc registered [12975.308692] Lustre: Echo OBD driver; http://www.lustre.org/ [12982.883036] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12982.918000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12984.348239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12988.625630] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [12994.534593] LustreError: 330636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12994.573465] LustreError: 330636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12997.683411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12998.050328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13002.267977] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13011.621893] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13018.797203] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13027.187979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13027.600849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13034.988017] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13043.805210] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13053.457621] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13054.948798] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13059.561099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13059.574162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13059.594908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13061.770357] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.16@tcp (stopping) [13063.657923] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13063.660509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13063.672325] Lustre: Skipped 2 previous similar messages [13066.834062] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.16@tcp (stopping) [13071.949984] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.16@tcp (stopping) [13071.956115] Lustre: Skipped 3 previous similar messages [13073.378314] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13073.748857] Lustre: server umount lustre-MDT0000 complete [13073.893878] LustreError: 331364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13073.914343] LustreError: 331364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13077.067339] LustreError: 332479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.16@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13077.785902] LustreError: 330616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788435531 with bad export cookie 17071409284209484300 [13077.790613] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13077.794284] LustreError: 330616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13079.014400] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13079.028282] Lustre: Skipped 1 previous similar message [13082.027869] LustreError: 330632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13082.028360] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13082.055351] LustreError: 330632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13082.060871] Lustre: Skipped 1 previous similar message [13087.201821] LustreError: 330631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13087.220868] LustreError: 330631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13092.320204] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13092.604254] Lustre: server umount lustre-MDT0001 complete [13096.262659] Lustre: server umount lustre-OST0000 complete [13105.804824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13111.228399] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13114.173879] Lustre: 334821:0:(mgs_llog.c:1814:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.116@tcp [13116.469348] Lustre: server umount MGS complete [13124.509997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13124.910797] LustreError: 335392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13124.928367] LustreError: 335392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13124.975451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13130.061625] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13137.879911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13138.258211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13143.075530] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13151.927641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13152.257581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13158.220222] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13166.043298] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13167.783826] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [13173.639980] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13176.378516] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [13183.050188] Lustre: server umount lustre-OST0000 complete [13189.609931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13189.617788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13189.627213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13189.635807] Lustre: Skipped 5 previous similar messages [13198.352744] Lustre: server umount lustre-MDT0000 complete [13199.328114] Lustre: 329948:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788435637/real 1788435637] req@000000007038521a x1875310465320320/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788435653 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13199.360288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13199.374420] Lustre: Skipped 1 previous similar message [13199.382838] LustreError: 335387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13199.397388] LustreError: 335387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13200.352636] Lustre: 329950:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788435638/real 1788435638] req@000000007c220cc7 x1875310465321088/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788435654 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13201.728107] LustreError: 338075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788435655 with bad export cookie 17071409284209485203 [13201.732259] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13201.736800] LustreError: 338075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [13202.183343] Lustre: server umount lustre-MDT0001 complete [13212.431659] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [13215.878306] Key type lgssc unregistered [13216.338580] LNet: 338826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13216.350598] LNetError: 338826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13217.388647] LNet: Removed LNI 192.168.204.116@tcp [13219.037083] Key type .llcrypt unregistered [13219.042194] Key type ._llcrypt unregistered [13240.834988] Key type ._llcrypt registered [13240.843149] Key type .llcrypt registered [13240.984926] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [13257.983970] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [13259.318933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13259.402650] alg: No test for adler32 (adler32-zlib) [13260.626228] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [13261.010839] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [13262.720576] Key type lgssc registered [13263.981716] Lustre: Echo OBD driver; http://www.lustre.org/ [13311.313488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13311.385419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13312.828474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13312.876967] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13312.997483] Lustre: lustre-MDT0000: new disk, initializing [13313.095233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13313.124682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13319.757448] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13333.786978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13333.913262] Lustre: 341989:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13334.074098] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13334.078912] Lustre: Skipped 1 previous similar message [13334.206697] Lustre: lustre-MDT0001: new disk, initializing [13334.314671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13334.390682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13334.403727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13339.815944] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13350.952866] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13358.121488] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13365.679971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13366.031496] Lustre: lustre-OST0000: new disk, initializing [13366.037425] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13366.045512] Lustre: 343960:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13366.154846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13372.902928] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13374.008678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13374.020228] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13374.144798] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13384.860786] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13390.855303] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13391.098223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13396.996604] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13397.262985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13400.034275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13400.037357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13400.045806] LustreError: Skipped 1 previous similar message [13400.060715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13405.159591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13405.166116] Lustre: Skipped 2 previous similar messages [13405.955754] Lustre: server umount lustre-OST0000 complete [13411.298478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13411.304796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13411.322262] Lustre: Skipped 1 previous similar message [13411.329986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13415.396069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13415.398183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13416.646562] Lustre: server umount lustre-MDT0000 complete [13421.047703] LustreError: 342794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788435874 with bad export cookie 16729113766510103582 [13421.049529] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13421.059390] LustreError: 342794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13421.461930] Lustre: server umount lustre-MDT0001 complete [13431.879468] Lustre: DEBUG MARKER: == conf-sanity test 163: dynamic MGS NIDs discovery ====== 07:44:43 (1788435883) [13434.122495] Lustre: DEBUG MARKER: === conf-sanity: start setup 07:44:45 (1788435885) === [13454.742391] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [13466.000649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13466.281717] LustreError: 347143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13466.299042] LustreError: 347143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13466.335563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13470.549512] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13471.726570] LustreError: 347144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13476.834787] LustreError: 347143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13480.199810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13480.717281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13486.413512] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13489.760050] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13497.393812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13497.666101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13505.383305] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13516.115040] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13516.352659] Lustre: lustre-OST0001: new disk, initializing [13516.356713] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [13516.369651] Lustre: 349454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13516.498102] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13522.654216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13522.681838] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13522.811742] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13523.643292] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13535.279492] Lustre: DEBUG MARKER: Using TIMEOUT=20 [13543.420965] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [13550.242525] Lustre: DEBUG MARKER: === conf-sanity: finish setup 07:46:41 (1788436001) === [13552.519963] Lustre: DEBUG MARKER: SKIP: conf-sanity test_163 Need multiple server nodes for FTM test [13558.256686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13558.265175] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13558.276567] LustreError: Skipped 1 previous similar message [13558.282610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13558.299607] Lustre: Skipped 1 previous similar message [13558.320529] Lustre: Skipped 2 previous similar messages [13562.769750] Lustre: server umount lustre-OST0000 complete [13563.882464] LustreError: 348630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13563.899599] LustreError: 348630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13567.971282] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13567.978671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13567.997319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13568.998264] LustreError: 348630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13573.552256] Lustre: server umount lustre-MDT0000 complete [13577.745526] LustreError: 349453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788436031 with bad export cookie 16729113766510104506 [13577.751550] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13577.765248] LustreError: 349453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13578.194795] Lustre: server umount lustre-MDT0001 complete [13592.639948] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [13595.616441] Lustre: 340208:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788436033/real 1788436033] req@ffff8b5d019aed80 x1875310767368448/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788436049 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [13595.665319] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13595.679735] Lustre: Skipped 2 previous similar messages [13598.688161] Lustre: 340211:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788436036/real 1788436036] req@ffff8b5e16490a80 x1875310767368576/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788436052 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [13600.737734] Lustre: 340210:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788436038/real 1788436038] req@ffff8b5e0ae86a00 x1875310767368704/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788436054 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [13603.808193] Lustre: 340208:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788436041/real 1788436041] req@ffff8b5d0cf80000 x1875310767368960/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788436057 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [13608.992515] Lustre: 340208:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788436046/real 1788436046] req@ffff8b5d0cf83100 x1875310767369088/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788436062 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [13671.918098] Lustre: server umount lustre-OST0001 complete [13680.578045] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [13698.495812] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [13738.904509] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 07:49:49 (1788436189) [13770.536228] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [13790.116774] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [13842.386046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13842.762777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13842.831212] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13843.103378] Lustre: lustre-MDT0000: new disk, initializing [13843.237411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13843.300265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13847.487430] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13858.704125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13858.924616] Lustre: 356793:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13859.083841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13859.100944] Lustre: Skipped 1 previous similar message [13859.251151] Lustre: lustre-MDT0001: new disk, initializing [13859.493702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13859.514673] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13867.038820] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13879.967883] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13888.875777] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13898.243044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13898.872542] Lustre: lustre-OST0000: new disk, initializing [13898.875196] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13898.902771] Lustre: 358764:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13899.911881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13899.928444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13900.189252] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13909.696884] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [13924.993408] Lustre: DEBUG MARKER: oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13971.940320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13971.948424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13971.972235] Lustre: Skipped 1 previous similar message [13971.980724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13971.985834] Lustre: Skipped 3 previous similar messages [13973.518817] Lustre: server umount lustre-OST0000 complete [13983.931153] Lustre: server umount lustre-MDT0000 complete [13987.298624] LustreError: 356806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13987.317848] LustreError: 356806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13987.641266] LustreError: 356785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788436441 with bad export cookie 16729113766510106046 [13987.648030] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13987.662753] LustreError: 356785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13988.047266] Lustre: server umount lustre-MDT0001 complete [13999.684334] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [14002.688551] Key type lgssc unregistered [14002.986264] LNet: 360942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14002.990956] LNetError: 360942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14003.007515] LNet: Removed LNI 192.168.204.116@tcp [14003.681182] Key type .llcrypt unregistered [14003.684437] Key type ._llcrypt unregistered [14105.820578] Key type ._llcrypt registered [14105.823489] Key type .llcrypt registered [14105.906257] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [14106.159318] Key type .llcrypt unregistered [14106.161736] Key type ._llcrypt unregistered [14192.898981] Key type ._llcrypt registered [14192.902736] Key type .llcrypt registered [14193.076700] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [14193.462543] Key type .llcrypt unregistered [14193.465189] Key type ._llcrypt unregistered [14230.746819] Key type ._llcrypt registered [14230.749479] Key type .llcrypt registered [14230.874944] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [14231.258871] Key type .llcrypt unregistered [14231.261787] Key type ._llcrypt unregistered [14269.171587] Key type ._llcrypt registered [14269.173202] Key type .llcrypt registered [14269.266383] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [14269.499354] Key type .llcrypt unregistered [14269.502656] Key type ._llcrypt unregistered [14358.164599] Key type ._llcrypt registered [14358.167605] Key type .llcrypt registered [14358.267731] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [14358.541656] Key type .llcrypt unregistered [14358.545624] Key type ._llcrypt unregistered [14393.737503] Key type ._llcrypt registered [14393.741699] Key type .llcrypt registered [14393.881697] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [14394.123618] Key type .llcrypt unregistered [14394.125897] Key type ._llcrypt unregistered [14432.238848] Key type ._llcrypt registered [14432.240692] Key type .llcrypt registered [14432.340488] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [14432.655695] Key type .llcrypt unregistered [14432.660182] Key type ._llcrypt unregistered [14468.317336] Key type ._llcrypt registered [14468.322846] Key type .llcrypt registered [14468.444607] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [14468.713606] Key type .llcrypt unregistered [14468.715481] Key type ._llcrypt unregistered [14505.983885] Key type ._llcrypt registered [14505.985680] Key type .llcrypt registered [14506.145376] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [14506.528451] Key type .llcrypt unregistered [14506.531147] Key type ._llcrypt unregistered [14545.645567] Key type ._llcrypt registered [14545.647977] Key type .llcrypt registered [14545.768190] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [14546.167329] Key type .llcrypt unregistered [14546.173120] Key type ._llcrypt unregistered [14607.259570] Key type ._llcrypt registered [14607.262509] Key type .llcrypt registered [14607.348510] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [14607.593456] Key type .llcrypt unregistered [14607.595245] Key type ._llcrypt unregistered [14641.149786] Key type ._llcrypt registered [14641.162144] Key type .llcrypt registered [14641.302380] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [14641.548536] Key type .llcrypt unregistered [14641.550982] Key type ._llcrypt unregistered [14680.618195] Key type ._llcrypt registered [14680.623674] Key type .llcrypt registered [14680.803223] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [14681.268339] Key type .llcrypt unregistered [14681.271956] Key type ._llcrypt unregistered [14734.449879] Key type ._llcrypt registered [14734.452643] Key type .llcrypt registered [14734.611838] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [14751.272225] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [14752.301264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14752.321361] alg: No test for adler32 (adler32-zlib) [14753.343762] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [14753.613839] LNet: Added LNI 192.168.204.116@tcp [8/256/0/180] [14755.280178] Key type lgssc registered [14756.392619] Lustre: Echo OBD driver; http://www.lustre.org/ [14793.244929] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14793.288644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14794.732226] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [14794.790606] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [14794.935572] Lustre: tst250fs-MDT0000: new disk, initializing [14795.056495] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14795.073110] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14800.546327] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [14811.814538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14811.961159] Lustre: 381348:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14812.108633] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [14812.118116] Lustre: Skipped 1 previous similar message [14812.205416] Lustre: tst250fs-MDT0001: new disk, initializing [14812.326894] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14812.404123] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14812.415388] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14816.959188] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [14821.764236] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [14829.140314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14829.685611] Lustre: tst250fs-OST0000: new disk, initializing [14829.692104] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [14829.703526] Lustre: 382981:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14829.843686] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14836.369094] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [14836.853486] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14836.865941] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14836.979231] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14847.828247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14848.024278] Lustre: tst250fs-OST0001: new disk, initializing [14848.029563] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14848.045994] Lustre: 383852:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14848.187229] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14853.389649] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14853.414265] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14853.611765] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14855.019166] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_default_debug -1 all [14860.449258] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14863.053564] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [14863.057465] Lustre: Skipped 2 previous similar messages [14884.326526] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14884.335302] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14884.335773] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14884.354141] Lustre: Skipped 2 previous similar messages [14884.360917] Lustre: Skipped 1 previous similar message [14886.953144] Lustre: server umount tst250fs-MDT0000 complete [14890.497251] LustreError: 381338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437344 with bad export cookie 15574708624027708486 [14890.503686] LustreError: mgc192.168.204.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14890.512848] LustreError: 381338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [14890.929698] Lustre: server umount tst250fs-MDT0001 complete [14904.261953] Lustre: server umount tst250fs-OST0000 complete [14918.140761] Lustre: server umount tst250fs-OST0001 complete [14936.883246] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [14946.777799] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [14984.231878] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 08:10:36 (1788437436) [14986.175478] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [15002.545385] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing set_hostid [15014.755736] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing load_modules_local [15047.389059] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14769 sec ========= 08:11:39 (1788437499) [15049.301703] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:11:41 (1788437501) === [15053.348238] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:11:45 (1788437505) === [15096.569625] Lustre: DEBUG MARKER: oleg416-server.virtnet: executing unload_modules_local [15099.505646] Key type lgssc unregistered [15099.870362] LNet: 393001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15099.882323] LNetError: 393001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15099.900078] LNet: Removed LNI 192.168.204.116@tcp [15100.739151] Key type .llcrypt unregistered [15100.745193] Key type ._llcrypt unregistered