[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 641581417 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002469] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004020] kvm-guest: setup PV IPIs [ 0.008890] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011140] LSM: Security Framework initializing [ 0.012105] Yama: becoming mindful. [ 0.013033] SELinux: Initializing. [ 0.014226] *** VALIDATE selinux *** [ 0.025407] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029990] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030188] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032046] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034080] *** VALIDATE tmpfs *** [ 0.036516] *** VALIDATE proc *** [ 0.037298] *** VALIDATE cgroup *** [ 0.038011] *** VALIDATE cgroup2 *** [ 0.039281] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040167] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042025] Spectre V2 : User space: Vulnerable [ 0.043012] Speculative Store Bypass: Vulnerable [ 0.048000] debug: unmapping init [mem 0xffffffffa6459000-0xffffffffa6460fff] [ 0.050630] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.053296] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.054032] ... version: 2 [ 0.055025] ... bit width: 48 [ 0.056018] ... generic registers: 4 [ 0.057017] ... value mask: 0000ffffffffffff [ 0.058014] ... max period: 00007fffffffffff [ 0.059017] ... fixed-purpose events: 3 [ 0.060015] ... event mask: 000000070000000f [ 0.061330] rcu: Hierarchical SRCU implementation. [ 0.064806] smp: Bringing up secondary CPUs ... [ 0.070893] x86: Booting SMP configuration: [ 0.071014] .... node #0, CPUs: #1 #2 #3 [ 0.111100] smp: Brought up 1 node, 4 CPUs [ 0.113013] smpboot: Max logical packages: 1 [ 0.114017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.187128] node 0 deferred pages initialised in 66ms [ 0.195014] devtmpfs: initialized [ 0.198272] x86/mm: Memory block size: 128MB [ 0.207223] gcov: version magic: 0x41383552 [ 0.212585] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.222109] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.228499] pinctrl core: initialized pinctrl subsystem [ 0.236235] [ 0.237025] ************************************************************* [ 0.241014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.247015] ** ** [ 0.254015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.259014] ** ** [ 0.266017] ** This means that this kernel is built to expose internal ** [ 0.267016] ** IOMMU data structures, which may compromise security on ** [ 0.277015] ** your system. ** [ 0.286014] ** ** [ 0.291018] ** If you see this message and you are not debugging the ** [ 0.302026] ** kernel, report this immediately to your vendor! ** [ 0.312015] ** ** [ 0.320014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.328014] ************************************************************* [ 0.343174] NET: Registered protocol family 16 [ 0.350503] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.361076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.370145] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.380504] cpuidle: using governor menu [ 0.381552] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.387535] PCI: Using configuration type 1 for base access [ 0.395168] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.432142] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.433000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.452044] cryptd: max_cpu_qlen set to 1000 [ 0.459313] ACPI: Added _OSI(Module Device) [ 0.460017] ACPI: Added _OSI(Processor Device) [ 0.461019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.462025] ACPI: Added _OSI(Processor Aggregator Device) [ 0.472666] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.494466] ACPI: Interpreter enabled [ 0.502066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.508031] ACPI: Using IOAPIC for interrupt routing [ 0.517084] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.527504] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.557032] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.565559] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.568025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.590080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.614234] acpiphp: Slot [2] registered [ 0.615000] acpiphp: Slot [5] registered [ 0.619051] acpiphp: Slot [6] registered [ 0.629156] acpiphp: Slot [7] registered [ 0.633170] acpiphp: Slot [8] registered [ 0.642188] acpiphp: Slot [9] registered [ 0.647325] acpiphp: Slot [10] registered [ 0.651186] acpiphp: Slot [3] registered [ 0.669171] acpiphp: Slot [4] registered [ 0.678133] acpiphp: Slot [11] registered [ 0.683178] acpiphp: Slot [12] registered [ 0.687176] acpiphp: Slot [13] registered [ 0.690000] acpiphp: Slot [14] registered [ 0.697140] acpiphp: Slot [15] registered [ 0.702024] acpiphp: Slot [16] registered [ 0.703000] acpiphp: Slot [17] registered [ 0.708167] acpiphp: Slot [18] registered [ 0.716177] acpiphp: Slot [19] registered [ 0.731820] acpiphp: Slot [20] registered [ 0.746145] acpiphp: Slot [21] registered [ 0.752181] acpiphp: Slot [22] registered [ 0.754139] acpiphp: Slot [23] registered [ 0.765500] acpiphp: Slot [24] registered [ 0.775502] acpiphp: Slot [25] registered [ 0.781157] acpiphp: Slot [26] registered [ 0.783155] acpiphp: Slot [27] registered [ 0.784000] acpiphp: Slot [28] registered [ 0.797256] acpiphp: Slot [29] registered [ 0.810713] acpiphp: Slot [30] registered [ 0.820037] acpiphp: Slot [31] registered [ 0.828074] PCI host bridge to bus 0000:00 [ 0.839029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.840034] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.852025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.866815] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.882026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.893030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.904229] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.919015] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.944423] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.990020] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 1.001230] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 1.010023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 1.016023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 1.028026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 1.038509] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 1.052641] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 1.061046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 1.067114] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x1e0 took 14648 usecs [ 1.074026] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 1.087015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 1.127015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 1.152015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 1.165177] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 1.187017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 1.214017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 1.259024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 1.285000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 1.301016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 1.321028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.367000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 1.394998] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 1.406000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 1.421016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 1.443017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 1.466016] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 1.478000] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 1.488016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 1.521014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 1.538675] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 1.545000] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 1.552015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 1.575016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 1.594483] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 1.607017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 1.619021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 1.657016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 1.674785] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.680745] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.684556] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.687639] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.691401] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.699038] iommu: Default domain type: Passthrough [ 1.704431] SCSI subsystem initialized [ 1.707797] ACPI: bus type USB registered [ 1.710168] usbcore: registered new interface driver usbfs [ 1.719085] usbcore: registered new interface driver hub [ 1.724090] usbcore: registered new device driver usb [ 1.727179] pps_core: LinuxPPS API ver. 1 registered [ 1.733011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.740093] PTP clock support registered [ 1.748706] EDAC MC: Ver: 3.0.0 [ 1.757193] PCI: Using ACPI for IRQ routing [ 1.759604] NetLabel: Initializing [ 1.760028] NetLabel: domain hash size = 128 [ 1.761014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.763093] NetLabel: unlabeled traffic allowed by default [ 1.765115] vgaarb: loaded [ 1.769282] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.771012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.787327] clocksource: Switched to clocksource kvm-clock [ 2.214055] VFS: Disk quotas dquot_6.6.0 [ 2.215452] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 2.223980] *** VALIDATE ramfs *** [ 2.226199] *** VALIDATE hugetlbfs *** [ 2.228888] pnp: PnP ACPI init [ 2.234643] pnp: PnP ACPI: found 6 devices [ 2.271964] hrtimer: interrupt took 10954302 ns [ 2.382081] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 2.405344] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 2.418558] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 2.426888] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 2.433479] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 2.447101] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 2.461968] NET: Registered protocol family 2 [ 2.473992] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 2.495380] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 2.503389] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 2.519184] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 2.531879] TCP: Hash tables configured (established 65536 bind 65536) [ 2.544288] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 2.564378] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.575330] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 2.586182] NET: Registered protocol family 1 [ 2.591148] RPC: Registered named UNIX socket transport module. [ 2.594100] RPC: Registered udp transport module. [ 2.595849] RPC: Registered tcp transport module. [ 2.597944] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 2.602029] NET: Registered protocol family 44 [ 2.606720] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 2.617360] pci 0000:00:00.0: quirk_natoma+0x0/0x20 took 10385 usecs [ 2.624126] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 2.628906] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 2.633048] PCI: CLS 0 bytes, default 64 [ 2.637298] Unpacking initramfs... [ 10.476617] debug: unmapping init [mem 0xffff93b7bcc54000-0xffff93b7bffbffff] [ 10.501220] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 10.507407] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 10.522841] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 13.768986] Initialise system trusted keyrings [ 13.780157] Key type blacklist registered [ 13.813627] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 13.889200] zbud: loaded [ 13.898574] *** VALIDATE nfs *** [ 13.918897] *** VALIDATE nfs4 *** [ 13.924078] pstore: using deflate compression [ 13.982177] Platform Keyring initialized [ 14.302360] NET: Registered protocol family 38 [ 14.304207] Key type asymmetric registered [ 14.318146] Asymmetric key parser 'x509' registered [ 14.325228] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 14.345287] io scheduler mq-deadline registered [ 14.347111] io scheduler kyber registered [ 14.360867] io scheduler bfq registered [ 14.365990] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 14.393195] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 14.398619] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 14.413625] ACPI: Power Button [PWRF] [ 14.442839] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 14.502927] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 14.600166] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 14.677191] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 14.771811] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 14.878773] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 14.973765] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 15.024652] Non-volatile memory driver v1.3 [ 15.026498] Linux agpgart interface v0.103 [ 15.280468] virtio_blk virtio1: [vda] 146216 512-byte logical blocks (74.9 MB/71.4 MiB) [ 15.311502] vda: detected capacity change from 0 to 74862592 [ 15.388545] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 15.412014] vdb: detected capacity change from 0 to 1073741824 [ 15.563751] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 15.585868] vdc: detected capacity change from 0 to 2621440000 [ 15.733285] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 15.754300] vdd: detected capacity change from 0 to 2621440000 [ 15.838563] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 15.847448] vde: detected capacity change from 0 to 4294967296 [ 15.965068] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 15.989868] vdf: detected capacity change from 0 to 4294967296 [ 16.074715] libphy: Fixed MDIO Bus: probed [ 16.157441] usbcore: registered new interface driver usbserial_generic [ 16.176670] usbserial: USB Serial support registered for generic [ 16.189553] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 16.218430] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 16.235118] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 16.252670] mousedev: PS/2 mouse device common for all mice [ 16.272257] rtc_cmos 00:05: RTC can wake from S4 [ 16.282739] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 16.294560] rtc_cmos 00:05: registered as rtc0 [ 16.294633] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 16.294677] intel_pstate: CPU model not supported [ 16.339731] hid: raw HID events driver (C) Jiri Kosina [ 16.339914] usbcore: registered new interface driver usbhid [ 16.339918] usbhid: USB HID core driver [ 16.340088] drop_monitor: Initializing network drop monitor service [ 16.340239] Initializing XFRM netlink socket [ 16.340624] NET: Registered protocol family 10 [ 16.345276] Segment Routing with IPv6 [ 16.345322] NET: Registered protocol family 17 [ 16.349927] mpls_gso: MPLS GSO support [ 16.478886] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 16.532847] RAS: Correctable Errors collector initialized. [ 16.535143] AVX version of gcm_enc/dec engaged. [ 16.549725] AES CTR mode by8 optimization enabled [ 16.568896] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 17.135736] sched_clock: Marking stable (17135690612, 0)->(20449142604, -3313451992) [ 17.158746] registered taskstats version 1 [ 17.186792] Loading compiled-in X.509 certificates [ 17.196718] zswap: loaded using pool lzo/zbud [ 17.394565] Key type big_key registered [ 17.516329] Key type encrypted registered [ 17.522899] ima: No TPM chip found, activating TPM-bypass! [ 17.537390] ima: Allocated hash algorithm: sha1 [ 17.556035] ima: No architecture policies found [ 17.571127] evm: Initialising EVM extended attributes: [ 17.575548] evm: security.selinux [ 17.586386] evm: security.ima [ 17.593525] evm: security.capability [ 17.605186] evm: HMAC attrs: 0x1 [ 17.609660] rtc_cmos 00:05: setting system clock to 2026-08-24 05:09:28 UTC (1787548168) [ 17.678169] debug: unmapping init [mem 0xffffffffa7403000-0xffffffffa75fffff] [ 17.742187] debug: unmapping init [mem 0xffffffffa6182000-0xffffffffa6458fff] [ 17.790026] Write protecting the kernel read-only data: 28672k [ 17.807545] debug: unmapping init [mem 0xffffffffa4803000-0xffffffffa49fffff] [ 17.836257] debug: unmapping init [mem 0xffffffffa5114000-0xffffffffa51fffff] [ 18.193745] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.214695] systemd[1]: Detected virtualization kvm. [ 18.218911] systemd[1]: Detected architecture x86-64. [ 18.223720] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 18.395228] systemd[1]: No hostname configured. [ 18.401594] systemd[1]: Set hostname to . [ 18.410265] random: systemd: uninitialized urandom read (16 bytes read) [ 18.430218] systemd[1]: Initializing machine ID from random generator. [ 18.891930] random: ln: uninitialized urandom read (6 bytes read) [ 19.359787] random: systemd: uninitialized urandom read (16 bytes read) [ 19.366613] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 19.444883] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 19.486336] urandom_read: 1 callbacks suppressed [ 19.486347] random: systemd: uninitialized urandom read (16 bytes read) [ 19.538068] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 19.566710] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Reached target Local File Systems. [ 19.623112] random: systemd: uninitialized urandom read (16 bytes read) Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 22.109110] device-mapper: uevent: version 1.0.3 [ 22.116823] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 24.082895] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon.[ 24.118583] virtio_net virtio0 ens2: renamed from eth0 [ 24.504605] scsi host0: ata_piix [ 24.767714] scsi host1: ata_piix [ 24.775119] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 24.778885] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 30.137890] random: crng init done [ 30.139544] random: 3 urandom warning(s) missed due to ratelimiting [ 32.453842] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 35.581485] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 40.511954] printk: systemd: 26 output lines suppressed due to ratelimiting [ 41.456600] SELinux: Disabled at runtime. [ 41.637984] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 41.662875] systemd[1]: Detected virtualization kvm. [ 41.670235] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 43.038202] systemd[1]: initrd-switch-root.service: Succeeded. [ 43.044758] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 43.058897] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 43.068308] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 43.077325] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 43.093268] systemd[1]: Starting Journal Service... Starting Journal Service... [ 43.134460] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... Startin[ 43.255790] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS g Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 45.599995] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 47.139023] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 47.531754] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 48.109358] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 48.287927] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit)[ 53.406695] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 53.878482] NFS: Registering the id_resolver key type [ 53.880163] Key type id_resolver registered [ 53.883036] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg446-server login: [ 110.348789] libcfs: loading out-of-tree module taints kernel. [ 110.464775] Key type ._llcrypt registered [ 110.472491] Key type .llcrypt registered [ 110.647601] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 138.171390] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 140.356537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 140.410543] alg: No test for adler32 (adler32-zlib) [ 142.465514] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 143.690672] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 145.607484] Key type lgssc registered [ 148.545641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 166.319370] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 210.065043] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 224.369534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 224.431884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 225.792746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 225.828948] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 225.952643] Lustre: lustre-MDT0000: new disk, initializing [ 226.124455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 226.175327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 231.770482] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.508683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 247.697763] Lustre: 6517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 247.728801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 247.733937] Lustre: Skipped 1 previous similar message [ 247.829787] Lustre: lustre-MDT0001: new disk, initializing [ 247.894424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 247.921783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 247.942471] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 253.070809] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.143633] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 269.386227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 269.784972] Lustre: lustre-OST0000: new disk, initializing [ 269.794471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 269.805127] Lustre: 8453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 269.917542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 273.766401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 273.786165] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 273.922327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 277.208809] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 293.062353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 293.214799] Lustre: lustre-OST0001: new disk, initializing [ 293.219369] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 293.224896] Lustre: 9529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 293.314549] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 300.317643] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 302.134430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 302.140578] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 302.211453] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 312.822489] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 322.001102] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 330.354612] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing check_logdir /tmp/testlogs/ [ 336.302656] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing yml_node [ 341.771983] Lustre: DEBUG MARKER: Client: 2.17.57.83 [ 344.959473] Lustre: DEBUG MARKER: MDS: 2.17.57.83 [ 347.752596] Lustre: DEBUG MARKER: OSS: 2.17.57.83 [ 349.596810] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 01:14:59 EDT 2026 [ 366.653252] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 368.204449] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 386.018515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 386.030892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 386.056482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 389.088264] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 389.089430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 389.107899] Lustre: Skipped 2 previous similar messages [ 389.119508] Lustre: Skipped 3 previous similar messages [ 393.434035] Lustre: server umount lustre-MDT0000 complete [ 397.142994] LustreError: 9528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787548548 with bad export cookie 7750198866438209275 [ 397.147421] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 397.149853] LustreError: 9528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 399.327796] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 399.330099] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 399.353348] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 399.411792] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 399.423820] Lustre: Skipped 1 previous similar message [ 402.338179] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 402.338276] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 402.345995] LustreError: 6523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 402.356620] Lustre: Skipped 1 previous similar message [ 403.520751] Lustre: server umount lustre-MDT0001 complete [ 408.151941] Lustre: server umount lustre-OST0000 complete [ 412.283230] Lustre: server umount lustre-OST0001 complete [ 420.954565] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 432.146617] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 472.085500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 472.320692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 472.342165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 472.401650] Lustre: lustre-MDT0000: new disk, initializing [ 472.463286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 472.483842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 477.454924] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 489.700450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.840144] Lustre: 15087:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 489.947243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 489.950710] Lustre: Skipped 1 previous similar message [ 490.030079] Lustre: lustre-MDT0001: new disk, initializing [ 490.156459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 490.215536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 490.227060] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 495.618403] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 505.926654] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 512.981704] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 520.709617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 521.115927] Lustre: lustre-OST0000: new disk, initializing [ 521.120126] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 521.130357] Lustre: 17057:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 521.262738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 522.465035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 522.491549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 522.633486] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 529.032877] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 538.436907] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 543.014856] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 543.237511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 548.976243] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 549.197538] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 553.445118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 553.464959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.468254] LustreError: Skipped 1 previous similar message [ 553.472605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 553.506579] Lustre: Skipped 2 previous similar messages [ 557.562681] Lustre: server umount lustre-OST0000 complete [ 562.145181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 562.160641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 562.185854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 562.188021] Lustre: Skipped 1 previous similar message [ 568.402751] Lustre: server umount lustre-MDT0000 complete [ 568.801278] LustreError: 15099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.821254] LustreError: 15099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 572.348425] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787548723 with bad export cookie 7750198866438210808 [ 572.358437] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 572.364445] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 572.851153] Lustre: server umount lustre-MDT0001 complete [ 579.309092] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 01:18:49 (1787548729) [ 599.009579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 599.403045] LustreError: 19933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.463508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 604.640610] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.913229] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 614.162741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 619.905559] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 627.893829] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 633.848870] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 635.974250] Lustre: Failing over lustre-MDT0000 [ 636.195116] Lustre: server umount lustre-MDT0000 complete [ 640.484921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 640.485201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 640.486882] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 640.486891] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 640.540810] Lustre: Skipped 2 previous similar messages [ 656.864523] Lustre: 3648:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787548791/real 1787548791] req@ffff93b83c119880 x1874380240091520/t0(0) o400->MGC192.168.204.146@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787548807 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 656.866616] LustreError: 20921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 656.935805] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 657.025289] LustreError: 20921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 659.554116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 667.419153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 667.427052] Lustre: Skipped 1 previous similar message [ 667.518905] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 672.745502] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 672.759767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 672.821134] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 673.371556] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 683.666270] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 688.102899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 688.119475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 688.144326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 688.149936] Lustre: Skipped 2 previous similar messages [ 693.930900] Lustre: server umount lustre-MDT0000 complete [ 697.315418] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 697.342097] LustreError: 19934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 698.727910] LustreError: 19914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787548849 with bad export cookie 7750198866438212222 [ 698.732781] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 698.742126] LustreError: 19914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 699.080966] Lustre: server umount lustre-MDT0001 complete [ 709.035039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.568475] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 721.388580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 725.637234] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 733.059782] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 739.987296] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 741.689403] Lustre: Failing over lustre-MDT0000 [ 741.700423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 741.701753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 741.705890] Lustre: Skipped 2 previous similar messages [ 741.709495] Lustre: Skipped 2 previous similar messages [ 741.881680] Lustre: server umount lustre-MDT0000 complete [ 761.833045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 762.146962] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 762.735599] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 767.712871] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 767.723324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 767.746134] Lustre: Skipped 1 previous similar message [ 767.788668] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 768.450797] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 778.584517] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 783.328222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 789.374100] Lustre: server umount lustre-MDT0000 complete [ 792.545354] LustreError: 23271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 792.559952] LustreError: 23271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 793.480469] LustreError: 24850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787548944 with bad export cookie 7750198866438213160 [ 793.494667] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 793.499044] LustreError: 24850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 793.793906] Lustre: server umount lustre-MDT0001 complete [ 801.875957] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 01:22:32 (1787548952) [ 818.395905] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 828.386782] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 872.815734] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 881.499514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 881.650516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 881.674218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 881.723976] Lustre: lustre-MDT0000: new disk, initializing [ 881.796145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 881.799581] Lustre: Skipped 3 previous similar messages [ 881.820438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 886.705881] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 899.569705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.729762] Lustre: 29651:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 900.196344] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 905.039847] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 914.824570] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 920.909750] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 927.776339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 928.230575] Lustre: 31621:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 929.476036] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 929.518756] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 929.613051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 929.627860] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 934.609740] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 944.273376] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 949.825463] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 970.720480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 970.732637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.748623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 970.751511] Lustre: Skipped 3 previous similar messages [ 970.768989] Lustre: Skipped 6 previous similar messages [ 973.761692] Lustre: server umount lustre-OST0000 complete [ 983.909838] Lustre: server umount lustre-MDT0000 complete [ 986.082612] LustreError: 29659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 986.107760] LustreError: 29659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 987.513594] LustreError: 30445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549138 with bad export cookie 7750198866438213566 [ 987.518589] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 987.530458] LustreError: 30445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 988.035988] Lustre: server umount lustre-MDT0001 complete [ 998.823469] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1001.411489] Key type lgssc unregistered [ 1001.733744] LNet: 33600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.755966] LNetError: 33600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1001.783186] LNet: Removed LNI 192.168.204.146@tcp [ 1002.629406] Key type .llcrypt unregistered [ 1002.632873] Key type ._llcrypt unregistered [ 1024.966938] Key type ._llcrypt registered [ 1024.973387] Key type .llcrypt registered [ 1025.099948] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1026.054553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1026.184599] alg: No test for adler32 (adler32-zlib) [ 1027.364592] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 1027.598336] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1029.279187] Key type lgssc registered [ 1030.293454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1038.253513] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1038.268866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.664155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1045.550599] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1050.086943] LustreError: 35015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1050.121374] LustreError: 35015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1055.201139] LustreError: 35016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1055.248741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.526627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1061.398536] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1070.068580] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1077.034191] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1084.754951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1085.173945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1091.683653] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1098.488812] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1100.564071] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1109.786986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1129.369355] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 01:27:58 (1787549278) [ 1131.699401] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1134.280129] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1144.640171] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 01:28:13 (1787549293) [ 1155.080337] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 01:28:25 (1787549305) [ 1156.185911] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1165.305382] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1165.310819] Lustre: Skipped 2 previous similar messages [ 1167.812381] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 01:28:37 (1787549317) [ 1169.667957] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1169.673193] Lustre: Skipped 2 previous similar messages [ 1178.433483] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1178.439640] Lustre: Skipped 4 previous similar messages [ 1192.477422] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 01:29:02 (1787549342) [ 1195.310835] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1198.622809] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1207.486534] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 01:29:17 (1787549357) [ 1209.364412] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1209.369968] Lustre: Skipped 6 previous similar messages [ 1225.226617] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 01:29:35 (1787549375) [ 1243.110154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1243.112191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1243.136905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1243.160529] Lustre: Skipped 1 previous similar message [ 1248.228610] LustreError: 36841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.249253] LustreError: 36841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1248.298166] Lustre: server umount lustre-OST0000 complete [ 1255.393093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1255.417930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.448756] Lustre: Skipped 1 previous similar message [ 1255.462944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.467642] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.469320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.502089] Lustre: Skipped 1 previous similar message [ 1261.308600] Lustre: server umount lustre-MDT0000 complete [ 1263.587369] LustreError: 37139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.641484] LustreError: 37139:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1265.810762] LustreError: 35750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549416 with bad export cookie 5865246176959807802 [ 1265.834358] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1265.835520] LustreError: 35750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1266.171351] Lustre: server umount lustre-MDT0001 complete [ 1278.352796] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1281.866942] Key type lgssc unregistered [ 1282.204726] LNet: 42899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1282.212146] LNetError: 42899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1282.225709] LNet: Removed LNI 192.168.204.146@tcp [ 1283.165936] Key type .llcrypt unregistered [ 1283.169271] Key type ._llcrypt unregistered [ 1303.890354] Key type ._llcrypt registered [ 1303.892654] Key type .llcrypt registered [ 1303.982661] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 1323.024739] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1324.882351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1325.368695] alg: No test for adler32 (adler32-zlib) [ 1326.989803] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 1327.300981] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1329.023146] Key type lgssc registered [ 1330.377890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1362.257547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1362.278873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1363.506608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1363.554219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1363.618492] Lustre: lustre-MDT0000: new disk, initializing [ 1363.678814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1363.697510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1369.690163] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1383.099248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.321242] Lustre: 46053:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1383.443113] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1383.457076] Lustre: Skipped 1 previous similar message [ 1383.636265] Lustre: lustre-MDT0001: new disk, initializing [ 1383.843484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1383.927163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1383.946612] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1389.908836] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1399.200778] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1404.763100] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1411.087308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1411.475057] Lustre: lustre-OST0000: new disk, initializing [ 1411.479407] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1411.488039] Lustre: 48023:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1411.581615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1418.486455] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1419.448612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1419.465937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1419.667991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1431.735561] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1438.325726] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1438.558233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1442.403513] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1442.710362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1445.346793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1445.350256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.353970] LustreError: Skipped 1 previous similar message [ 1445.355254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1445.360666] Lustre: Skipped 1 previous similar message [ 1450.465967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1450.478106] Lustre: Skipped 2 previous similar messages [ 1455.590342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1455.609683] Lustre: Skipped 1 previous similar message [ 1459.168120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1459.479743] Lustre: server umount lustre-OST0000 complete [ 1465.827121] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1465.830620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1465.848428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1465.862848] Lustre: Skipped 1 previous similar message [ 1469.834282] Lustre: server umount lustre-MDT0000 complete [ 1473.321520] LustreError: 46045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549624 with bad export cookie 10136748428234655199 [ 1473.332421] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1473.334834] LustreError: 46045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1473.654226] Lustre: server umount lustre-MDT0001 complete [ 1482.434708] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 01:33:52 (1787549632) [ 1488.763910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.154246] LustreError: 50486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.179451] LustreError: 50486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1489.234139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1493.796907] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1494.501262] LustreError: 50487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1499.618155] LustreError: 50486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1502.221681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1502.623984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1507.402417] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1514.391853] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1519.943445] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1526.238273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1526.696929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1533.294476] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1541.847196] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1546.519962] LustreError: 53014:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1547.669071] Lustre: Modifying parameter general.timeout=1 in log params [ 1548.745540] Lustre: Modifying parameter general.timeout=2 in log params [ 1550.829238] Lustre: Modifying parameter general.timeout=4 in log params [ 1550.833967] Lustre: Skipped 1 previous similar message [ 1555.180771] Lustre: Modifying parameter general.timeout=9 in log params [ 1555.184810] Lustre: Skipped 4 previous similar messages [ 1563.400014] Lustre: Modifying parameter general.timeout=19 in log params [ 1563.412147] Lustre: Skipped 9 previous similar messages [ 1580.056934] Lustre: Modifying parameter general.timeout=38 in log params [ 1580.061959] Lustre: Skipped 18 previous similar messages [ 1612.537532] Lustre: Modifying parameter general.timeout=74 in log params [ 1612.542114] Lustre: Skipped 35 previous similar messages [ 1670.157567] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 01:37:00 (1787549820) [ 1673.103675] Lustre: 59953:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1675.083658] Lustre: 60057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1677.952681] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 in log params [ 1677.959786] Lustre: Skipped 60 previous similar messages [ 1679.002758] Lustre: 60261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1679.017352] Lustre: 60261:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1709.394122] Lustre: 61835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1709.402165] Lustre: 61835:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1709.422297] LustreError: 61835:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1713.283640] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 01:37:43 (1787549863) [ 1727.968496] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1727.982582] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1728.012804] Lustre: Skipped 1 previous similar message [ 1728.036231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1731.553379] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1734.052873] Lustre: server umount lustre-MDT0000 complete [ 1736.677622] LustreError: 50487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1736.697456] LustreError: 50487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1737.641261] LustreError: 50466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787549888 with bad export cookie 10136748428234656088 [ 1737.646990] LustreError: 50466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1737.649132] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1738.058495] Lustre: server umount lustre-MDT0001 complete [ 1752.378972] Lustre: server umount lustre-OST0000 complete [ 1780.528331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.969334] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1781.249290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1784.857651] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1794.563443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1794.748471] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1794.810070] Lustre: 64367:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1801.864128] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1812.368578] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1819.669152] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1827.498740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1827.798561] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1828.190292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1828.196233] Lustre: Skipped 1 previous similar message [ 1834.973234] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1843.004065] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1844.516183] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1844.523890] Lustre: Skipped 18 previous similar messages [ 1844.567878] Lustre: 66932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1855.458594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1855.467154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1855.481562] Lustre: Skipped 1 previous similar message [ 1855.485864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1855.488541] Lustre: Skipped 3 previous similar messages [ 1858.051718] Lustre: server umount lustre-OST0000 complete [ 1862.112464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1862.122252] LustreError: Skipped 1 previous similar message [ 1866.209062] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1866.230743] Lustre: Skipped 2 previous similar messages [ 1868.128822] Lustre: server umount lustre-MDT0000 complete [ 1871.341786] LustreError: 64379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1871.395832] LustreError: 64379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1872.829198] LustreError: 66934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550023 with bad export cookie 10136748428234659112 [ 1872.839674] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1872.847631] LustreError: 66934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1873.393599] Lustre: server umount lustre-MDT0001 complete [ 1885.900034] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 1889.186361] Key type lgssc unregistered [ 1889.552151] LNet: 68143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.576475] LNetError: 68143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.606110] LNet: Removed LNI 192.168.204.146@tcp [ 1890.714410] Key type .llcrypt unregistered [ 1890.715617] Key type ._llcrypt unregistered [ 1920.125876] Key type ._llcrypt registered [ 1920.128140] Key type .llcrypt registered [ 1920.245254] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 1921.723317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1921.782919] alg: No test for adler32 (adler32-zlib) [ 1922.936231] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 1923.115994] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 1924.791205] Key type lgssc registered [ 1926.274492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1933.956548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1933.991300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1935.616088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.428617] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1946.086201] LustreError: 69558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1946.115833] LustreError: 69558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1949.717063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1950.516893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1955.314723] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1964.100304] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1970.279305] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1978.399976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1979.209075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1986.581515] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 1993.428384] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2001.490919] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2011.108638] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.115984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2011.118329] Lustre: Skipped 2 previous similar messages [ 2011.124916] Lustre: Skipped 2 previous similar messages [ 2016.238828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2016.257432] Lustre: Skipped 2 previous similar messages [ 2017.395854] Lustre: server umount lustre-MDT0000 complete [ 2020.543218] LustreError: 69539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550171 with bad export cookie 2267687525071391063 [ 2020.559528] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2020.563342] LustreError: 69539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2020.877092] Lustre: server umount lustre-MDT0001 complete [ 2035.043786] Lustre: server umount lustre-OST0000 complete [ 2067.185030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.534893] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2067.578113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2067.878953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.821696] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2082.094421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2082.281152] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2082.336037] Lustre: 74430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2082.881223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2087.936534] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2096.135655] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2101.502437] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2107.741354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2107.863922] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2108.200519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.533388] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2122.809206] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2124.527124] Lustre: 77006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2126.415833] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2130.916453] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2130.920362] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.936389] LustreError: Skipped 1 previous similar message [ 2130.951740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2130.957607] Lustre: Skipped 1 previous similar message [ 2136.034082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2136.039442] Lustre: Skipped 1 previous similar message [ 2136.981960] Lustre: server umount lustre-OST0000 complete [ 2144.224649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2144.234246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2144.248535] Lustre: Skipped 1 previous similar message [ 2144.255103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2146.274787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.001089] Lustre: server umount lustre-MDT0000 complete [ 2151.393388] LustreError: 74441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.413040] LustreError: 74441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2151.954099] LustreError: 75188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550302 with bad export cookie 2267687525071391910 [ 2151.962399] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2151.970699] LustreError: 75188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2152.493079] Lustre: server umount lustre-MDT0001 complete [ 2165.686394] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2168.758712] Key type lgssc unregistered [ 2169.223374] LNet: 78219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2169.235215] LNetError: 78219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2169.284589] LNet: Removed LNI 192.168.204.146@tcp [ 2170.441295] Key type .llcrypt unregistered [ 2170.443136] Key type ._llcrypt unregistered [ 2194.482496] Key type ._llcrypt registered [ 2194.486743] Key type .llcrypt registered [ 2194.719046] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2196.215239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2196.238136] alg: No test for adler32 (adler32-zlib) [ 2197.358541] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 2197.617520] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2199.327169] Key type lgssc registered [ 2200.930775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2211.124784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2211.151362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2212.730463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2217.149721] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2223.079812] LustreError: 79632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2223.110057] LustreError: 79632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2226.020114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2226.652881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2231.991241] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2239.614075] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2245.857281] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2253.117299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2253.796760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2260.880720] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2269.024654] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2278.091893] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2279.205544] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2281.189518] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2281.199097] Lustre: Skipped 1 previous similar message [ 2283.965602] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2283.977914] Lustre: Skipped 2 previous similar messages [ 2288.401318] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2288.406640] Lustre: Skipped 4 previous similar messages [ 2296.558689] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2296.569940] Lustre: Skipped 7 previous similar messages [ 2312.795978] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2312.801518] Lustre: Skipped 15 previous similar messages [ 2345.654541] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2345.658103] Lustre: Skipped 36 previous similar messages [ 2380.369712] Lustre: 87741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2384.142825] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 01:48:53 (1787550533) [ 2386.584909] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2388.615917] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 01:48:58 (1787550538) [ 2394.089768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2394.104250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2394.105234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2394.131839] Lustre: Skipped 1 previous similar message [ 2399.213454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2399.224868] Lustre: Skipped 2 previous similar messages [ 2399.598735] Lustre: server umount lustre-OST0000 complete [ 2405.856767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2405.869480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.899565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2409.443364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2409.445185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2409.466691] Lustre: Skipped 1 previous similar message [ 2411.899721] Lustre: server umount lustre-MDT0000 complete [ 2414.560517] LustreError: 79628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2414.579422] LustreError: 79628:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2417.001725] LustreError: 82916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550567 with bad export cookie 4077838852662501883 [ 2417.017312] LustreError: 82916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2417.018600] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2417.478408] Lustre: server umount lustre-MDT0001 complete [ 2430.322209] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2433.903841] Key type lgssc unregistered [ 2434.215828] LNet: 88994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2434.222701] LNetError: 88994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2434.240230] LNet: Removed LNI 192.168.204.146@tcp [ 2435.242488] Key type .llcrypt unregistered [ 2435.248109] Key type ._llcrypt unregistered [ 2441.224516] Key type ._llcrypt registered [ 2441.227212] Key type .llcrypt registered [ 2441.367872] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2447.280300] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules [ 2448.532267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2448.810337] alg: No test for adler32 (adler32-zlib) [ 2449.968925] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 2450.325953] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2452.031187] Key type lgssc registered [ 2453.014679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2454.400613] LustreError: 90245:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2473.514387] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2477.000174] LustreError: 90245:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2484.939214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2484.963848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2486.656524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2491.756330] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2496.992562] LustreError: 91265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2497.023513] LustreError: 91265:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2500.065512] LustreError: 91266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2500.354695] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 01:50:50 (1787550650) [ 2505.223074] LustreError: 91265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.887701] LustreError: 91297:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2506.908451] LustreError: 91297:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2506.942294] LustreError: 91297:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2508.311892] Lustre: server umount lustre-MDT0000 complete [ 2528.514066] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 2532.149364] Key type lgssc unregistered [ 2532.445589] LNet: 92794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2532.465454] LNetError: 92794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2532.539862] LNet: Removed LNI 192.168.204.146@tcp [ 2534.384488] Key type .llcrypt unregistered [ 2534.385958] Key type ._llcrypt unregistered [ 2556.595392] Key type ._llcrypt registered [ 2556.597416] Key type .llcrypt registered [ 2556.766676] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2558.045618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2558.158253] alg: No test for adler32 (adler32-zlib) [ 2559.262143] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 2559.508529] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 2561.167204] Key type lgssc registered [ 2562.790552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2570.794748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2570.819025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2572.401525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2577.828511] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2583.010619] LustreError: 94012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2583.034237] LustreError: 94012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2587.028646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.739504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2592.741790] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2600.406914] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2605.623848] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2611.620270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2612.369285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2620.082362] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2630.229644] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2685.481366] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 01:53:55 (1787550835) [ 2687.351910] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2689.578538] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 01:53:59 (1787550839) [ 2694.636416] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.638261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2694.658245] Lustre: Skipped 2 previous similar messages [ 2694.684583] Lustre: Skipped 2 previous similar messages [ 2699.754547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.758362] Lustre: Skipped 2 previous similar messages [ 2700.278873] Lustre: server umount lustre-MDT0000 complete [ 2704.553191] LustreError: 93993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550855 with bad export cookie 8774128601160598579 [ 2704.562742] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2704.591349] LustreError: 93993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2704.864992] LustreError: 94008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.870192] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2704.999952] Lustre: server umount lustre-MDT0001 complete [ 2711.741261] Lustre: server umount lustre-OST0000 complete [ 2725.739463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2726.160798] LustreError: 98312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.179466] LustreError: 98312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2726.313413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2731.133523] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2731.492848] LustreError: 98313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2736.620175] LustreError: 98312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2740.089817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2741.228075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2744.930773] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2753.254645] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2760.470342] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2774.378041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2774.709243] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2774.716747] LustreError: 98303:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2774.741133] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2774.757304] LustreError: 100396:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2774.776384] LustreError: 100396:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2774.791800] LustreError: 100396:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2774.946169] Lustre: server umount lustre-OST0000 complete [ 2774.949275] LustreError: 100396:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2783.704083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2784.016744] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2784.030098] LustreError: 98302:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2784.056973] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2784.066729] LustreError: 100862:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -98 [ 2784.075973] LustreError: 100862:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 2784.082306] LustreError: 100862:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2784.149502] Lustre: server umount lustre-OST0000 complete [ 2784.152527] LustreError: 100862:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2794.242973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2794.518811] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2794.533289] Lustre: Found index 0 for lustre-OST0000, updating log [ 2794.546412] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2794.679120] Lustre: lustre-OST0000: new disk, initializing [ 2794.689070] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2794.702523] Lustre: 101396:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2795.050600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.183848] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2805.821517] Lustre: Failing over lustre-OST0000 [ 2805.943361] Lustre: server umount lustre-OST0000 complete [ 2810.528616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2810.549727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2810.568756] Lustre: Skipped 1 previous similar message [ 2815.969189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2815.977700] Lustre: Skipped 1 previous similar message [ 2820.180605] Lustre: server umount lustre-MDT0000 complete [ 2821.090975] LustreError: 99370:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.117397] LustreError: 99370:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2824.358215] LustreError: 98294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787550975 with bad export cookie 8774128601160601323 [ 2824.362200] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2824.369577] LustreError: 98294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2824.830661] Lustre: server umount lustre-MDT0001 complete [ 2833.924572] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 01:56:23 (1787550983) [ 2851.216817] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 2861.756530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2862.246536] LustreError: 103743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2862.391735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2866.863282] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2876.073812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2880.660488] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2883.566196] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2891.280597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2892.064798] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2898.776964] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2901.224101] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2901.234701] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:00 [ 2901.248529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2906.598407] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:54 [ 2906.637938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2908.684394] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:02 [ 2908.712072] Lustre: lustre-OST0000: Recovery over after 0:07, of 1 clients 1 recovered and 0 were evicted. [ 2908.723929] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2908.745776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2908.755289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2908.822141] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2908.888870] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2909.029096] Lustre: lustre-OST0001: new disk, initializing [ 2909.032663] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2909.039514] Lustre: 106058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2909.493365] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2909.507337] Lustre: Skipped 2 previous similar messages [ 2917.267172] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2918.134655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2918.144500] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2918.409499] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2926.823582] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2935.337538] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2941.496763] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [ 2943.460055] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2943.479460] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.511866] Lustre: Skipped 1 previous similar message [ 2946.065464] Lustre: server umount lustre-MDT0001 complete [ 2946.553875] LustreError: 103738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2946.572302] LustreError: 103738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2957.476417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2957.545272] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2957.552791] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2957.582758] Lustre: 103733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2959.168541] LustreError: 107599:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2959.186102] LustreError: 107599:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2959.209119] LustreError: 103735:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2959.400299] LustreError: 107609:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.146@tcp: cfg command failed: rc = -17 [ 2959.429360] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2959.429360] [ 2963.969673] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2964.007168] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 2974.074924] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 01:58:43 (1787551123) [ 3000.781217] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3019.389118] Lustre: 109699:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3042.268854] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3045.940921] Lustre: 110833:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3137.506883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3137.515383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3137.523632] Lustre: Skipped 2 previous similar messages [ 3137.529760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3137.536382] Lustre: Skipped 3 previous similar messages [ 3139.515870] Lustre: server umount lustre-MDT0000 complete [ 3143.149708] LustreError: 107178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3143.180396] LustreError: 107178:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 3143.780681] LustreError: 103724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787551294 with bad export cookie 8774128601160602324 [ 3143.789426] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3143.805550] LustreError: 103724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3144.365710] Lustre: server umount lustre-MDT0001 complete [ 3159.558347] Lustre: server umount lustre-OST0000 complete [ 3174.052097] Lustre: server umount lustre-OST0001 complete [ 3181.930240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3194.413418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3203.811495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3212.759814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3239.795658] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3252.964501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3252.979527] Lustre: lustre-MDT0000: reset Object Index mappings [ 3253.480298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3253.485987] Lustre: Skipped 1 previous similar message [ 3259.231854] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3267.526299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3267.559776] Lustre: lustre-MDT0001: reset Object Index mappings [ 3273.441230] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3276.740211] Lustre: 115948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3284.478832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3291.561832] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3292.587716] LustreError: 116300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3292.605138] LustreError: 116300:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3292.630644] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3298.803895] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 3301.201960] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3307.515452] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:195 to 0x2c0000400:225) [ 3307.526134] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 3309.455789] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3320.311257] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3324.826684] Lustre: 117826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3377.284428] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 02:05:27 (1787551527) [ 3404.771256] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3404.786453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3404.789534] Lustre: Skipped 5 previous similar messages [ 3404.814813] Lustre: Skipped 7 previous similar messages [ 3410.442938] Lustre: server umount lustre-MDT0000 complete [ 3413.494982] LustreError: 115551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787551564 with bad export cookie 8774128601160649420 [ 3413.508700] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3413.517482] LustreError: 115551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3420.209837] Lustre: server umount lustre-MDT0001 complete [ 3431.266960] Lustre: server umount lustre-OST0000 complete [ 3436.032192] LustreError: 118784:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000400:0x3f:0x0].0x0 (ffff93b706137e00) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3450.335468] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3450.554169] Lustre: server umount lustre-OST0001 complete [ 3457.096640] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 3469.253319] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3518.573639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3519.218861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3519.350130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3519.560942] Lustre: lustre-MDT0000: new disk, initializing [ 3519.773107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3519.792801] Lustre: Skipped 3 previous similar messages [ 3519.859178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3527.651023] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3536.279743] Lustre: server umount lustre-MDT0000 complete [ 3548.713345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3549.167056] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3549.188308] Lustre: 122099:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3549.202400] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3555.445954] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3562.360990] Lustre: server umount lustre-MDT0000 complete [ 3572.742352] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 02:08:42 (1787551722) [ 3574.658894] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3577.024424] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 02:08:46 (1787551726) [ 3597.801954] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 3608.653122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.186751] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3623.378350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3623.552400] Lustre: 124149:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3623.629110] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3623.638884] Lustre: Skipped 1 previous similar message [ 3623.759642] Lustre: lustre-MDT0001: new disk, initializing [ 3623.914717] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3623.935230] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3629.268697] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3634.431255] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3641.673886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3641.931956] Lustre: lustre-OST0000: new disk, initializing [ 3641.940545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3641.947689] Lustre: 125727:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3644.072596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3644.095312] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3644.243420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3648.373798] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3661.610117] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3661.841129] Lustre: lustre-OST0001: new disk, initializing [ 3661.846749] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3661.854473] Lustre: 126599:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3663.823971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3663.839069] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3664.038134] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3670.191812] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 3681.085061] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3684.888844] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4124.086940] Lustre: 133023:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4124.097849] Lustre: 133023:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4124.105202] Lustre: 133023:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4124.113320] Lustre: 133023:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4124.123893] Lustre: 133023:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4124.132759] Lustre: 133023:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4433.826322] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 02:23:02 (1787552582) [ 4452.344961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4452.351386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4452.369899] Lustre: Skipped 5 previous similar messages [ 4452.370985] Lustre: Skipped 9 previous similar messages [ 4455.912835] Lustre: server umount lustre-MDT0000 complete [ 4457.442639] LustreError: 124156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4457.461115] LustreError: 124156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 4460.011267] LustreError: 124142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787552610 with bad export cookie 8774128601160717418 [ 4460.019165] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4460.027307] LustreError: 124142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4462.562443] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4462.572575] Lustre: Skipped 1 previous similar message [ 4462.576591] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4462.581626] Lustre: Skipped 2 previous similar messages [ 4466.677927] Lustre: server umount lustre-MDT0001 complete [ 4477.467193] Lustre: server umount lustre-OST0000 complete [ 4488.041660] Lustre: server umount lustre-OST0001 complete [ 4496.131385] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 4507.765610] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 4544.499577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4544.768895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4544.814076] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4544.898560] Lustre: lustre-MDT0000: new disk, initializing [ 4544.996619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4545.010618] Lustre: Skipped 5 previous similar messages [ 4545.036979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4550.858978] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4561.714052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4561.882044] Lustre: 136089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4561.969774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4561.985784] Lustre: Skipped 1 previous similar message [ 4562.090701] Lustre: lustre-MDT0001: new disk, initializing [ 4562.270028] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4562.296511] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4567.497729] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4577.947627] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4584.533817] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4591.608705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4591.917141] Lustre: lustre-OST0000: new disk, initializing [ 4591.921360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4591.935954] Lustre: 138056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4592.050751] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4592.064342] Lustre: Skipped 1 previous similar message [ 4593.623103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4593.640124] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4593.738112] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4597.996227] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 4609.014038] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4620.790673] Lustre: lustre-MDD0000: changelog on [ 4624.461694] Lustre: lustre-MDD0001: changelog on [ 6412.306554] Lustre: lustre-MDD0001: changelog off [ 6414.114853] Lustre: 137214:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6415.013495] Lustre: lustre-MDD0000: changelog off [ 6420.980143] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6420.985288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6420.994135] Lustre: Skipped 1 previous similar message [ 6420.999334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6421.003109] Lustre: Skipped 2 previous similar messages [ 6424.546674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6424.555250] Lustre: Skipped 2 previous similar messages [ 6426.574579] Lustre: server umount lustre-OST0000 complete [ 6431.200904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6431.208774] LustreError: Skipped 1 previous similar message [ 6431.225106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6431.245914] Lustre: Skipped 1 previous similar message [ 6431.267811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6434.792244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6437.260477] Lustre: server umount lustre-MDT0000 complete [ 6439.913652] LustreError: 136100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6439.940172] LustreError: 136100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6441.369109] LustreError: 136080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554592 with bad export cookie 8774128601161541346 [ 6441.375208] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6441.377625] LustreError: 136080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6441.759821] Lustre: server umount lustre-MDT0001 complete [ 6455.111941] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 6457.786463] Key type lgssc unregistered [ 6458.002838] LNet: 147181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6458.009539] LNetError: 147181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6458.030029] LNet: Removed LNI 192.168.204.146@tcp [ 6458.847164] Key type .llcrypt unregistered [ 6458.849078] Key type ._llcrypt unregistered [ 6479.110606] Key type ._llcrypt registered [ 6479.114304] Key type .llcrypt registered [ 6479.245040] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 6494.824721] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6495.808594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6495.954337] alg: No test for adler32 (adler32-zlib) [ 6496.998464] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 6497.273273] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 6498.999186] Key type lgssc registered [ 6499.980917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6534.404350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6534.419077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6535.655686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6535.709904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6535.873320] Lustre: lustre-MDT0000: new disk, initializing [ 6536.076876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6536.124072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6540.585738] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6552.333134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6552.495468] Lustre: 150385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6552.585612] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6552.593185] Lustre: Skipped 1 previous similar message [ 6552.693064] Lustre: lustre-MDT0001: new disk, initializing [ 6552.868404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6552.942529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6552.953335] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6559.041978] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6569.942192] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6575.760687] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6581.963618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6582.328166] Lustre: lustre-OST0000: new disk, initializing [ 6582.330654] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6582.336653] Lustre: 152354:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6582.417473] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6588.023174] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6589.023649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6589.040119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6589.144382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6596.592813] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6601.512196] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_module obdecho/obdecho [ 6603.394456] LustreError: 153356:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6603.399501] LustreError: 153356:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6605.028028] LustreError: 153453:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6614.497489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6614.501942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6614.528743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6615.009796] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6615.022026] Lustre: Skipped 1 previous similar message [ 6615.028197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6615.038038] Lustre: Skipped 2 previous similar messages [ 6620.130518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6620.148760] Lustre: Skipped 2 previous similar messages [ 6620.626613] Lustre: server umount lustre-MDT0000 complete [ 6624.165838] LustreError: 150378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554775 with bad export cookie 17395283099294777736 [ 6624.173730] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6624.195194] LustreError: 150378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6624.530616] Lustre: server umount lustre-MDT0001 complete [ 6638.832465] Lustre: server umount lustre-OST0000 complete [ 6659.658966] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 6672.485106] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 6707.493253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6707.763126] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6707.834946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6707.898237] Lustre: lustre-MDT0000: new disk, initializing [ 6708.004315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6708.030954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6713.013856] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6722.641155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6722.785186] Lustre: 157170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6722.853743] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6722.857772] Lustre: Skipped 1 previous similar message [ 6722.905134] Lustre: lustre-MDT0001: new disk, initializing [ 6722.988969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6723.030689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6723.047028] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6726.057793] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6734.193220] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6740.940577] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6747.881213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6748.180745] Lustre: lustre-OST0000: new disk, initializing [ 6748.189680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6748.201485] Lustre: 159146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6748.383477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6754.186690] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6754.357693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6754.367597] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6754.509919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6762.721215] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6766.454977] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6766.647126] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6770.910066] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6771.208942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6775.265655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6775.267772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6775.275285] LustreError: Skipped 1 previous similar message [ 6775.280139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6775.299018] Lustre: Skipped 1 previous similar message [ 6779.425735] Lustre: server umount lustre-OST0000 complete [ 6784.481744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6784.493310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6784.509413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6784.511601] Lustre: Skipped 1 previous similar message [ 6795.744609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6795.755474] Lustre: Skipped 5 previous similar messages [ 6797.279186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6797.465477] Lustre: server umount lustre-MDT0000 complete [ 6800.783030] LustreError: 157162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787554951 with bad export cookie 17395283099294778695 [ 6800.784975] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6800.792852] LustreError: 157162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6801.078230] Lustre: server umount lustre-MDT0001 complete [ 6804.403189] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 03:02:34 (1787554954) [ 6806.188535] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6807.920169] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 03:02:38 (1787554958) [ 6814.679816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6815.094290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6818.989400] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6820.324343] LustreError: 161558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6820.340062] LustreError: 161558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6824.426277] LustreError: 161559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6826.743534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6831.562296] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6838.725526] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6843.814808] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6850.306780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6850.705486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6850.712840] Lustre: Skipped 1 previous similar message [ 6857.921574] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 6866.261230] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6873.057480] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6873.062808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6873.070216] Lustre: Skipped 1 previous similar message [ 6873.072994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6878.940275] Lustre: server umount lustre-MDT0000 complete [ 6880.739984] LustreError: 162287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6880.750977] LustreError: 162287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6882.141134] LustreError: 162295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555033 with bad export cookie 17395283099294779577 [ 6882.156327] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6882.157553] LustreError: 162295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6885.856964] LustreError: 161554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6885.857677] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6885.857687] Lustre: Skipped 2 previous similar messages [ 6885.897226] LustreError: 161554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6888.725571] Lustre: server umount lustre-MDT0001 complete [ 6894.511241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6908.108447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6908.347245] LustreError: 165398:0:(llog.c:1655:llog_backup()) MGC192.168.204.146@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6908.353781] Lustre: 165398:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6912.482303] LustreError: 165403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6912.508581] LustreError: 165403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6922.722167] LustreError: 165405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6922.745867] LustreError: 165405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 6942.709396] LustreError: 165404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6942.732939] LustreError: 165404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 6969.311284] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6969.323751] Lustre: 165398:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.146@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6978.022097] LustreError: 165405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6978.034000] LustreError: 165405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 25 previous similar messages [ 7010.297286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7014.670069] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7015.402559] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7023.171806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7027.682836] LustreError: 148603:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d8c8f1ef x1874386902708480/t0(0) o250->MGC192.168.204.146@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7032.353199] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7033.336612] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7039.865869] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7041.625957] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7046.590780] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7048.093795] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7069.160185] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7069.169432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7069.171099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7069.193703] Lustre: Skipped 2 previous similar messages [ 7069.210757] Lustre: Skipped 10 previous similar messages [ 7072.945834] Lustre: server umount lustre-MDT0000 complete [ 7074.273725] LustreError: 165432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7074.293352] LustreError: 165432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 7076.357829] LustreError: 162295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555227 with bad export cookie 17395283099294786031 [ 7076.368269] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7076.634120] Lustre: server umount lustre-MDT0001 complete [ 7090.744221] Lustre: server umount lustre-OST0000 complete [ 7112.677473] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7123.990573] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7161.120097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7161.320798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7161.377464] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7161.509922] Lustre: lustre-MDT0000: new disk, initializing [ 7161.634945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7161.646353] Lustre: Skipped 1 previous similar message [ 7161.684184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7166.930373] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7178.930631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7179.044304] Lustre: 170503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7179.124275] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7179.425855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7185.287884] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7196.063899] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7203.100581] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7210.079664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7210.536739] Lustre: lustre-OST0000: new disk, initializing [ 7210.541121] Lustre: Skipped 1 previous similar message [ 7210.548388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7210.556269] Lustre: Skipped 2 previous similar messages [ 7210.562287] Lustre: 172475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7218.067391] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7218.270932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7218.293153] Lustre: Skipped 1 previous similar message [ 7218.304216] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7218.509198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7229.255388] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7235.102177] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7235.300116] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7240.371108] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7240.764745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7244.257340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7244.259533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7244.293691] Lustre: Skipped 1 previous similar message [ 7249.325531] Lustre: server umount lustre-OST0000 complete [ 7259.585472] Lustre: server umount lustre-MDT0000 complete [ 7259.618474] LustreError: 170516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7259.647482] LustreError: 170516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7263.742957] LustreError: 171314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555414 with bad export cookie 17395283099294802866 [ 7263.746041] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7263.759645] LustreError: 171314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7264.201932] Lustre: server umount lustre-MDT0001 complete [ 7273.241968] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 03:10:23 (1787555423) [ 7279.705060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7285.978735] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7295.861060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7301.831841] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7309.811135] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7316.672914] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7325.092238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7332.193772] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7340.129800] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7354.267675] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 03:11:43 (1787555503) [ 7362.530287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7362.533930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7362.544652] Lustre: Skipped 2 previous similar messages [ 7362.545684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7362.545690] Lustre: Skipped 4 previous similar messages [ 7365.108783] Lustre: server umount lustre-OST0000 complete [ 7375.697741] Lustre: server umount lustre-MDT0000 complete [ 7379.999172] LustreError: 174866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555530 with bad export cookie 17395283099294803748 [ 7380.006186] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7380.007994] LustreError: 174866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7380.331262] Lustre: server umount lustre-MDT0001 complete [ 7393.498558] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7397.150402] Key type lgssc unregistered [ 7397.558794] LNet: 178621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7397.565887] LNetError: 178621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7397.582215] LNet: Removed LNI 192.168.204.146@tcp [ 7398.585067] Key type .llcrypt unregistered [ 7398.587120] Key type ._llcrypt unregistered [ 7421.079979] Key type ._llcrypt registered [ 7421.081649] Key type .llcrypt registered [ 7421.188273] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7422.731673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7422.825155] alg: No test for adler32 (adler32-zlib) [ 7424.128314] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 7424.518961] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7426.279461] Key type lgssc registered [ 7427.879984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7438.638550] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7438.684487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7454.495466] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7459.615333] LustreError: 179863:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.146@tcp: failed processing log, type 4: rc = -110 [ 7464.735900] LustreError: 179863:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7464.746561] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7464.764071] LustreError: MGC192.168.204.146@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7464.785681] LustreError: 179863:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7464.798691] LustreError: 179863:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -22 [ 7464.805150] LustreError: 179863:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7464.900880] Lustre: server umount lustre-OST0000 complete [ 7464.907530] LustreError: 179863:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7475.209906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7476.861876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.289649] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7486.434834] LustreError: 180362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7486.468243] LustreError: 180362:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7491.558401] LustreError: 180363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7491.991270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7492.599454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7498.259615] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7508.128551] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7515.776654] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7523.962679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7524.207695] LustreError: 182180:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7524.214246] Lustre: 182180:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.146@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7524.317363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7531.309698] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7539.447583] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7543.790641] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7543.796083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7543.799388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7543.814248] Lustre: Skipped 1 previous similar message [ 7548.626410] Lustre: server umount lustre-MDT0000 complete [ 7548.897417] LustreError: 180363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7548.922957] LustreError: 180363:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7552.869185] LustreError: 182208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555703 with bad export cookie 6092463123836873461 [ 7552.882703] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7553.263826] Lustre: server umount lustre-MDT0001 complete [ 7567.388670] Lustre: server umount lustre-OST0000 complete [ 7577.629880] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7590.309227] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7631.298793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7631.536511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7631.565958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7631.667609] Lustre: lustre-MDT0000: new disk, initializing [ 7631.789436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7631.812618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7636.763254] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7648.936836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7649.125987] Lustre: 185708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7649.196450] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7649.199520] Lustre: Skipped 1 previous similar message [ 7649.276249] Lustre: lustre-MDT0001: new disk, initializing [ 7649.387515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7649.452902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7649.466535] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7655.104977] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7666.768203] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7673.941051] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7682.190715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7682.543332] Lustre: lustre-OST0000: new disk, initializing [ 7682.552742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7682.560688] Lustre: 187678:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7682.655755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7689.884609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7689.895186] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7690.084118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7690.562295] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7701.371148] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7706.703807] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7707.004488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7712.502626] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7712.751363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7715.814401] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7715.817221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7715.828930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7715.842277] Lustre: Skipped 2 previous similar messages [ 7715.859845] Lustre: Skipped 3 previous similar messages [ 7720.945037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7720.958030] Lustre: Skipped 1 previous similar message [ 7726.049517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7726.073833] Lustre: Skipped 1 previous similar message [ 7729.119817] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7729.434473] Lustre: server umount lustre-OST0000 complete [ 7736.291313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7736.296312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7736.308696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7736.310238] Lustre: Skipped 1 previous similar message [ 7736.348849] Lustre: Skipped 1 previous similar message [ 7740.366341] Lustre: server umount lustre-MDT0000 complete [ 7741.409104] LustreError: 185718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7741.444625] LustreError: 185718:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7744.472155] LustreError: 185700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787555895 with bad export cookie 6092463123836874175 [ 7744.477334] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7744.479208] LustreError: 185700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7744.939866] Lustre: server umount lustre-MDT0001 complete [ 7755.149617] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 03:18:24 (1787555904) [ 7775.352732] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 7778.763636] Key type lgssc unregistered [ 7779.078742] LNet: 190630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7779.088561] LNetError: 190630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7779.105597] LNet: Removed LNI 192.168.204.146@tcp [ 7780.049707] Key type .llcrypt unregistered [ 7780.052353] Key type ._llcrypt unregistered [ 7800.371324] Key type ._llcrypt registered [ 7800.373601] Key type .llcrypt registered [ 7800.503527] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7801.947128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7801.993262] alg: No test for adler32 (adler32-zlib) [ 7803.086382] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 7803.337558] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 7805.039244] Key type lgssc registered [ 7806.424339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7815.368305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7815.386219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7816.688192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7821.597948] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7826.928987] LustreError: 191851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7826.949868] LustreError: 191851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7831.805491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7832.035090] LustreError: 191850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7832.485311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7838.766881] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7848.446752] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7855.237843] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7863.141465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7863.362870] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7863.369982] LustreError: 193716:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7863.388805] Lustre: 193716:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.146@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7863.569342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7870.372896] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7879.186298] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7883.234348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7883.240943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7883.263897] Lustre: Skipped 2 previous similar messages [ 7883.279771] Lustre: Skipped 2 previous similar messages [ 7888.205796] Lustre: server umount lustre-MDT0000 complete [ 7888.365839] LustreError: 192779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7888.389636] LustreError: 192779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7892.583605] LustreError: 191831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556043 with bad export cookie 1229983604669546211 [ 7892.602221] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7892.620518] LustreError: 191831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7893.152411] Lustre: server umount lustre-MDT0001 complete [ 7908.145079] Lustre: server umount lustre-OST0000 complete [ 7918.524739] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 7930.097176] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 7976.435391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7976.930513] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7976.997114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7977.190174] Lustre: lustre-MDT0000: new disk, initializing [ 7977.320231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7977.357596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7982.942611] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 7994.109093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7994.325307] Lustre: 197239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7994.387897] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7994.390374] Lustre: Skipped 1 previous similar message [ 7994.395224] LustreError: 198024:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7994.400674] LustreError: 198024:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7994.404812] Lustre: 198024:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7994.422291] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7994.428947] Lustre: Skipped 1 previous similar message [ 7994.567759] Lustre: lustre-MDT0001: new disk, initializing [ 7994.734671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7994.773175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7994.794719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7999.661313] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8009.911779] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8016.065660] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8022.611775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8022.883879] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8022.885960] Lustre: Skipped 1 previous similar message [ 8022.888169] LustreError: 199186:0:(llog.c:1696:llog_backup()) MGC192.168.204.146@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8022.905987] LustreError: 199186:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8022.914238] Lustre: 199186:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.146@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8022.925627] Lustre: 199186:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8022.966582] Lustre: lustre-OST0000: new disk, initializing [ 8022.982354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8022.991555] Lustre: 199209:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8023.101097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8029.723302] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8031.806514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8031.819376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8031.937407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8039.671747] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8044.165343] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8044.361550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8049.501580] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8049.746809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8052.716188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8052.716675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8052.731354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8052.748936] Lustre: Skipped 1 previous similar message [ 8052.792610] LustreError: Skipped 1 previous similar message [ 8057.839113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8057.853265] Lustre: Skipped 2 previous similar messages [ 8058.167841] Lustre: server umount lustre-OST0000 complete [ 8062.945244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8062.948565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8062.961032] Lustre: Skipped 1 previous similar message [ 8062.981448] Lustre: Skipped 1 previous similar message [ 8068.067523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8068.086793] Lustre: Skipped 1 previous similar message [ 8068.821131] Lustre: server umount lustre-MDT0000 complete [ 8073.199912] LustreError: 197252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8073.233071] LustreError: 197252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8073.421252] LustreError: 197233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556224 with bad export cookie 1229983604669546925 [ 8073.424994] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8073.436667] LustreError: 197233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8073.816906] Lustre: server umount lustre-MDT0001 complete [ 8082.730109] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 03:23:53 (1787556233) [ 8100.376596] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8110.578848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8110.876501] LustreError: 202337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8110.944648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8116.103883] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8120.297214] LustreError: 202337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8120.348433] LustreError: 202337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8125.228137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8130.538101] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8133.764510] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8141.462705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8149.187673] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8158.424261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8158.663507] Lustre: lustre-OST0001: new disk, initializing [ 8158.668871] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8158.676671] Lustre: 204649:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8158.793654] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8158.802912] Lustre: Skipped 2 previous similar messages [ 8165.685476] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8168.071836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8168.079560] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8168.154380] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8177.364673] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8189.866758] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8192.215162] Lustre: DEBUG MARKER: ADD OST3 [ 8198.020895] loop: module loaded [ 8198.109849] loop0: detected capacity change from 0 to 204800000 [ 8198.397508] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8198.418600] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8198.439058] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8204.850699] loop0: detected capacity change from 0 to 204800000 [ 8205.782789] Lustre: DEBUG MARKER: STOP OST3 [ 8211.931797] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8212.234899] Lustre: lustre-OST0003: new disk, initializing [ 8212.247028] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8212.259496] Lustre: 206680:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8213.925280] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8214.066184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8214.083952] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8214.120948] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8215.969732] LustreError: 206809:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8215.988954] Lustre: Failing over lustre-OST0003 [ 8216.366368] Lustre: server umount lustre-OST0003 complete [ 8219.129594] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8219.139143] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8219.141102] LustreError: 203823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8219.141113] LustreError: 203823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8219.152248] Lustre: Skipped 1 previous similar message [ 8220.056905] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8228.816637] Lustre: DEBUG MARKER: START OST3 again [ 8237.328993] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8237.665363] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8237.678214] Lustre: Skipped 1 previous similar message [ 8237.695547] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8238.907831] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8239.282111] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8239.284055] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8245.805334] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8268.509815] Lustre: Failing over lustre-OST0003 [ 8268.692673] Lustre: server umount lustre-OST0003 complete [ 8270.305515] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8270.315686] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8270.343385] LustreError: 203825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8270.362603] LustreError: 203825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8288.744571] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8288.752566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8288.768172] Lustre: Skipped 3 previous similar messages [ 8288.786473] Lustre: Skipped 3 previous similar messages [ 8293.663285] Lustre: server umount lustre-MDT0000 complete [ 8298.967387] LustreError: 204648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556449 with bad export cookie 1229983604669547807 [ 8299.005592] LustreError: 204648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8299.014262] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8305.576157] Lustre: server umount lustre-MDT0001 complete [ 8311.383878] Lustre: server umount lustre-OST0000 complete [ 8317.083539] Lustre: server umount lustre-OST0001 complete [ 8326.493219] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8336.639269] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8380.160874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8380.461590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8380.489133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8380.615483] Lustre: lustre-MDT0000: new disk, initializing [ 8380.726468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8380.755596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8386.730635] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8399.995565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8400.200302] Lustre: 212109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8400.282945] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8400.289214] Lustre: Skipped 1 previous similar message [ 8400.409462] Lustre: lustre-MDT0001: new disk, initializing [ 8400.607632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8400.628890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8405.780279] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8415.095382] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8421.672616] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8430.082853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8430.472876] Lustre: 214079:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8432.148610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8432.331069] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8438.686941] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8450.900594] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8456.171055] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8456.582971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8462.544952] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8462.983096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8468.454219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8468.461831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8468.463674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8468.463682] Lustre: Skipped 3 previous similar messages [ 8468.468082] LustreError: Skipped 1 previous similar message [ 8468.490944] Lustre: Skipped 4 previous similar messages [ 8480.223191] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8480.699453] Lustre: server umount lustre-OST0000 complete [ 8487.912980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8491.881594] Lustre: server umount lustre-MDT0000 complete [ 8494.054386] LustreError: 212975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8494.094732] LustreError: 212975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 8496.647626] LustreError: 212102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556647 with bad export cookie 1229983604669550103 [ 8496.652070] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8496.657850] LustreError: 212102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8496.999157] Lustre: server umount lustre-MDT0001 complete [ 8501.973642] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 03:30:51 (1787556651) [ 8524.064601] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8537.143131] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8578.545718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8578.776594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8578.833380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8578.843870] Lustre: Skipped 1 previous similar message [ 8578.963924] Lustre: lustre-MDT0000: new disk, initializing [ 8578.966530] Lustre: Skipped 1 previous similar message [ 8579.031828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8579.050388] Lustre: Skipped 1 previous similar message [ 8585.337255] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8597.270184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8597.444434] Lustre: 218693:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8597.814131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8604.282509] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8616.104192] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8624.251673] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8632.885354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8633.354984] Lustre: 220663:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8635.206161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8635.419312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8641.097256] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8651.950142] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8657.936930] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8658.214687] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8663.795734] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8664.153705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8671.206075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8671.208612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8671.226109] LustreError: Skipped 1 previous similar message [ 8671.228466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8671.228473] Lustre: Skipped 8 previous similar messages [ 8671.260070] Lustre: Skipped 3 previous similar messages [ 8672.707594] Lustre: server umount lustre-OST0000 complete [ 8683.835801] Lustre: server umount lustre-MDT0000 complete [ 8686.560701] LustreError: 219552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8686.570708] LustreError: 219552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8688.539447] LustreError: 218684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556839 with bad export cookie 1229983604669550985 [ 8688.543373] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8688.559370] LustreError: 218684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8689.038490] Lustre: server umount lustre-MDT0001 complete [ 8699.104974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8699.465933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8699.476427] Lustre: Skipped 5 previous similar messages [ 8705.079681] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8716.056505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8722.252583] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8731.884447] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8738.819082] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8749.290546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8757.187535] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 8767.154312] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8779.744342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8779.749849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8779.757782] LustreError: Skipped 1 previous similar message [ 8779.764549] Lustre: Skipped 2 previous similar messages [ 8779.766628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8779.776105] Lustre: Skipped 5 previous similar messages [ 8783.543623] Lustre: server umount lustre-OST0000 complete [ 8794.836974] Lustre: server umount lustre-MDT0000 complete [ 8800.081763] LustreError: 222863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787556950 with bad export cookie 1229983604669551867 [ 8800.105479] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8800.107497] LustreError: 222863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8800.651558] Lustre: server umount lustre-MDT0001 complete [ 8908.151478] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 8911.940629] Key type lgssc unregistered [ 8912.430267] LNet: 226552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8912.441244] LNetError: 226552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8912.475845] LNet: Removed LNI 192.168.204.146@tcp [ 8913.609169] Key type .llcrypt unregistered [ 8913.615977] Key type ._llcrypt unregistered [ 8943.602259] Key type ._llcrypt registered [ 8943.605690] Key type .llcrypt registered [ 8943.711686] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 8963.264618] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 8964.279280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8964.447757] alg: No test for adler32 (adler32-zlib) [ 8966.021786] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 8966.562086] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 8968.399266] Key type lgssc registered [ 8970.326823] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9019.295312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9019.322387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9020.667597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9020.716327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9020.815357] Lustre: lustre-MDT0000: new disk, initializing [ 9020.895380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9020.924926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9026.202714] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9039.126182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9039.221625] Lustre: 229908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9039.317920] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9039.324993] Lustre: Skipped 1 previous similar message [ 9039.418300] Lustre: lustre-MDT0001: new disk, initializing [ 9039.513504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9039.544548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9039.552183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9044.457918] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9054.533901] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9060.520287] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9068.771325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9069.142038] Lustre: lustre-OST0000: new disk, initializing [ 9069.145943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9069.158419] Lustre: 231881:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9069.260059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9076.334723] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9077.349281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9077.365947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9077.529795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9086.443306] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9091.640385] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9091.891959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9096.944388] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9097.214135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9103.368020] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9103.373367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9103.394103] LustreError: Skipped 1 previous similar message [ 9103.400548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9103.434230] Lustre: Skipped 1 previous similar message [ 9105.657235] Lustre: server umount lustre-OST0000 complete [ 9111.522782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9111.531259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9111.562945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9111.578632] Lustre: Skipped 1 previous similar message [ 9113.569863] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9113.570130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9113.606017] Lustre: Skipped 1 previous similar message [ 9115.649420] Lustre: server umount lustre-MDT0000 complete [ 9119.610025] LustreError: 229902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557270 with bad export cookie 13406733630186791916 [ 9119.631602] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9119.633100] LustreError: 229902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9120.056348] Lustre: server umount lustre-MDT0001 complete [ 9140.888884] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9151.693263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9152.165019] LustreError: 234814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9152.193756] LustreError: 234814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9152.243800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9156.945544] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9157.600434] LustreError: 234815:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9162.723031] LustreError: 234814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9166.152984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9166.582411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9171.842940] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9175.574485] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9183.334380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9183.658810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9190.717994] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9200.586720] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9200.777084] Lustre: lustre-OST0001: new disk, initializing [ 9200.781519] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9200.788346] Lustre: 237122:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9200.872301] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9208.538529] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9209.049746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9209.064060] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9209.211476] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9219.222971] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9227.179970] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9243.525512] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 03:43:13 (1787557393) [ 9248.739135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9248.747217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9248.760935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9251.907587] Lustre: server umount lustre-MDT0000 complete [ 9255.404681] LustreError: 235542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9255.425810] LustreError: 235542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9255.820698] LustreError: 234796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557406 with bad export cookie 13406733630186792805 [ 9255.826856] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9255.841775] LustreError: 234796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9256.133341] Lustre: server umount lustre-MDT0001 complete [ 9271.001229] Lustre: server umount lustre-OST0000 complete [ 9285.644319] Lustre: server umount lustre-OST0001 complete [ 9291.465904] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9300.440659] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9341.337231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9341.702700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9341.732893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9341.821462] Lustre: lustre-MDT0000: new disk, initializing [ 9341.927881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9341.943651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9350.847289] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9364.294159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9364.386212] Lustre: 241072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9364.448225] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9364.452237] Lustre: Skipped 1 previous similar message [ 9364.525321] Lustre: lustre-MDT0001: new disk, initializing [ 9364.680556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9364.692099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9370.832933] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9381.497674] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9387.454848] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9395.141490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9395.460588] Lustre: lustre-OST0000: new disk, initializing [ 9395.467183] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9395.477918] Lustre: 243040:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9396.779507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9396.793520] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9396.979691] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9402.506679] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9412.471125] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9417.999197] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9418.247922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9423.778202] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9424.081928] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9427.944234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9427.946182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9427.968222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9427.968233] Lustre: Skipped 4 previous similar messages [ 9427.979979] Lustre: Skipped 4 previous similar messages [ 9428.018508] LustreError: Skipped 1 previous similar message [ 9432.581324] Lustre: server umount lustre-OST0000 complete [ 9438.179959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9438.198041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9438.204820] Lustre: Skipped 2 previous similar messages [ 9443.299885] Lustre: server umount lustre-MDT0000 complete [ 9443.300281] LustreError: 241084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9443.338589] LustreError: 241084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9447.634533] LustreError: 241063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557598 with bad export cookie 13406733630186794940 [ 9447.646668] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9447.659200] LustreError: 241063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9448.153109] Lustre: server umount lustre-MDT0001 complete [ 9457.602028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9458.048343] LustreError: 245258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9458.061534] LustreError: 245258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9458.137296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9458.145409] Lustre: Skipped 2 previous similar messages [ 9463.171918] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9473.054031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9479.465129] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9487.782646] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9494.303858] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9501.911714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9508.357980] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9516.593633] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9667.582518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9667.594926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9667.602085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9667.641057] Lustre: Skipped 2 previous similar messages [ 9677.280037] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9677.537566] Lustre: server umount lustre-OST0000 complete [ 9696.223274] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9696.811832] Lustre: server umount lustre-MDT0000 complete [ 9697.252511] LustreError: 246211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9697.274332] LustreError: 246211:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9701.040116] LustreError: 245238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787557851 with bad export cookie 13406733630186795822 [ 9701.043831] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9701.049665] LustreError: 245238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9701.591940] Lustre: server umount lustre-MDT0001 complete [ 9787.302258] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [ 9790.564885] Key type lgssc unregistered [ 9790.831202] LNet: 248875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9790.843121] LNetError: 248875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9791.917847] LNet: Removed LNI 192.168.204.146@tcp [ 9792.861308] Key type .llcrypt unregistered [ 9792.866303] Key type ._llcrypt unregistered [ 9822.890368] Key type ._llcrypt registered [ 9822.893956] Key type .llcrypt registered [ 9823.054103] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [ 9838.589547] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9839.569022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9839.591339] alg: No test for adler32 (adler32-zlib) [ 9840.619193] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [ 9840.855204] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [ 9842.591181] Key type lgssc registered [ 9843.898769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9901.851723] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [ 9914.099614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9914.212707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9915.638744] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9915.696131] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9915.817828] Lustre: lustre-MDT0000: new disk, initializing [ 9915.932507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9916.026677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9921.372438] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9933.476706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9933.622911] Lustre: 252965:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9933.713931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9933.717976] Lustre: Skipped 1 previous similar message [ 9933.805365] Lustre: lustre-MDT0001: new disk, initializing [ 9933.879663] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9933.923367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9933.929639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9939.930247] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9945.360705] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9953.427937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9953.918726] Lustre: lustre-OST0000: new disk, initializing [ 9953.932216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9953.958930] Lustre: 254599:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9954.188563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9959.526073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9959.533679] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9959.683961] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9962.060293] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9973.185314] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9973.319281] Lustre: lustre-OST0001: new disk, initializing [ 9973.323320] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9973.328451] Lustre: 255471:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9973.422384] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9980.043046] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [ 9983.107261] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9983.120407] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9983.279820] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9992.232531] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10004.500490] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10019.297686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10019.308971] Lustre: Skipped 1 previous similar message [10019.310268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10023.501022] Lustre: server umount lustre-MDT0000 complete [10027.316167] LustreError: 252960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558178 with bad export cookie 1675319114216851354 [10027.334436] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10027.341893] LustreError: 252960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10027.783253] Lustre: server umount lustre-MDT0001 complete [10043.080984] Lustre: server umount lustre-OST0000 complete [10046.047958] Lustre: 250451:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787558180/real 1787558180] req@ffff93b7121b9f80 x1874390408051584/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787558196 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10046.089307] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10047.253341] Lustre: server umount lustre-OST0001 complete [10064.234945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10075.051064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10083.976606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10094.044283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10112.012367] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10120.510362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10120.526935] Lustre: lustre-MDT0000: reset Object Index mappings [10120.835758] LustreError: 260482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10120.850164] LustreError: 260482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10120.898061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10124.613312] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10126.309348] LustreError: 260483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10130.409456] LustreError: 260482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10133.612544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10133.631511] Lustre: lustre-MDT0001: reset Object Index mappings [10133.978919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10138.220690] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10141.016495] Lustre: 261623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10149.142053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10156.145518] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10158.774589] LustreError: 261978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10164.207948] LustreError: 261977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10164.392182] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10164.651742] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10164.664467] Lustre: Skipped 1 previous similar message [10165.688763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10165.713484] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10170.492425] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10177.565433] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10181.209886] Lustre: 263493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10186.215901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10186.224241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10186.234852] Lustre: Skipped 2 previous similar messages [10186.256199] Lustre: Skipped 6 previous similar messages [10187.943064] Lustre: server umount lustre-MDT0000 complete [10191.253503] LustreError: 260462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558342 with bad export cookie 1675319114216858144 [10191.266325] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10191.267152] LustreError: 260462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10191.328704] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10191.329432] LustreError: 260478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10191.354859] Lustre: Skipped 1 previous similar message [10191.362663] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10191.367065] Lustre: Skipped 1 previous similar message [10191.608773] Lustre: server umount lustre-MDT0001 complete [10197.207773] Lustre: server umount lustre-OST0000 complete [10200.428158] Lustre: server umount lustre-OST0001 complete [10218.056939] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 03:59:28 (1787558368) [10233.955181] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [10244.302356] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10279.018385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10279.271988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10279.318354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10279.424935] Lustre: lustre-MDT0000: new disk, initializing [10279.505482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10279.538567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10283.454251] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10293.303543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10293.482469] Lustre: 267745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10293.556225] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10293.565879] Lustre: Skipped 1 previous similar message [10293.784593] Lustre: lustre-MDT0001: new disk, initializing [10294.086320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10294.105767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10298.408689] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10309.414074] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10316.989025] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10323.519944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10323.736435] Lustre: lustre-OST0000: new disk, initializing [10323.740459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10323.747283] Lustre: 269724:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10325.303699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10325.315683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10325.416082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10330.234424] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10339.967936] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10344.738955] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10344.929914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10349.616722] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10349.839329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10356.193488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10356.197783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10356.212328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10356.215063] Lustre: Skipped 2 previous similar messages [10356.229032] Lustre: Skipped 1 previous similar message [10357.761750] Lustre: server umount lustre-OST0000 complete [10361.329608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10361.338633] Lustre: Skipped 1 previous similar message [10367.912838] Lustre: server umount lustre-MDT0000 complete [10371.569844] LustreError: 267757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10371.591855] LustreError: 267757:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10371.940960] LustreError: 267738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558522 with bad export cookie 1675319114216860237 [10371.953611] LustreError: 267738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10371.958867] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10372.360950] Lustre: server umount lustre-MDT0001 complete [10391.226436] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10403.788578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10404.288995] LustreError: 272658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10404.350818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10404.357592] Lustre: Skipped 2 previous similar messages [10409.465842] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10418.882382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10423.550333] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10426.833975] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10435.115727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10443.644289] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10456.649418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10456.894075] Lustre: lustre-OST0001: new disk, initializing [10456.897946] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10456.903226] Lustre: 274968:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10458.405984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10458.415830] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10458.673951] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10463.910862] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10474.194637] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10477.993891] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10484.674231] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10484.846269] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10495.974930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10495.986106] LustreError: Skipped 1 previous similar message [10495.994158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10496.023190] Lustre: Skipped 2 previous similar messages [10496.042561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10496.054954] Lustre: Skipped 2 previous similar messages [10497.583693] Lustre: server umount lustre-MDT0000 complete [10499.561257] LustreError: 272658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10499.599562] LustreError: 272658:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10502.053589] LustreError: 272638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558652 with bad export cookie 1675319114216861119 [10502.069210] LustreError: 272638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10502.078404] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10502.526714] Lustre: server umount lustre-MDT0001 complete [10518.177239] Lustre: server umount lustre-OST0000 complete [10521.057102] Lustre: 250451:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787558655/real 1787558655] req@ffff93b70cc37100 x1874390408209664/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787558671 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10521.101356] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10521.118951] Lustre: Skipped 3 previous similar messages [10522.800606] Lustre: server umount lustre-OST0001 complete [10540.835470] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10553.246798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10553.726438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10553.735434] Lustre: Skipped 3 previous similar messages [10560.323440] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10572.758526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10578.664534] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10582.324731] Lustre: 279166:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10590.051040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10596.895917] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10598.658853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10606.815972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10612.794354] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10617.187442] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10630.573791] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10636.944625] Lustre: 281046:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10644.083852] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10644.106595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10655.273652] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 04:06:44 (1787558804) [10674.148545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10674.154342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10674.168691] Lustre: Skipped 1 previous similar message [10674.185873] Lustre: Skipped 3 previous similar messages [10685.407699] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10685.638322] Lustre: server umount lustre-MDT0000 complete [10689.087125] LustreError: 279544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787558839 with bad export cookie 1675319114216867657 [10689.090399] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10689.100745] LustreError: 279544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10689.505479] LustreError: 279216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10689.524068] LustreError: 279216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10695.517817] Lustre: server umount lustre-MDT0001 complete [10705.114865] Lustre: server umount lustre-OST0000 complete [10708.198191] Lustre: server umount lustre-OST0001 complete [10715.189573] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [10727.392453] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10783.025443] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [10796.131124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10796.407498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10796.454725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10796.574446] Lustre: lustre-MDT0000: new disk, initializing [10796.667843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10802.643748] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10815.253093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10815.474617] Lustre: 285291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10815.761548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10815.767221] Lustre: Skipped 4 previous similar messages [10815.811302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10822.063583] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10833.827945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10834.104129] Lustre: 286924:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10835.543177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10841.635775] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10854.078903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10854.392465] Lustre: 287794:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10856.311414] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10856.319378] Lustre: Skipped 1 previous similar message [10856.432166] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10861.714568] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10871.421818] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10875.821039] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10875.826680] Lustre: Skipped 1 previous similar message [10892.032479] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 04:10:42 (1787559042) [10896.298927] Lustre: server umount lustre-MDT0000 complete [10897.386111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10897.399324] Lustre: Skipped 7 previous similar messages [10899.561165] LustreError: 285284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559050 with bad export cookie 1675319114216875364 [10899.564498] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10899.578510] LustreError: 285284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10900.120765] Lustre: server umount lustre-MDT0001 complete [10913.404674] Lustre: server umount lustre-OST0000 complete [10927.843419] Lustre: server umount lustre-OST0001 complete [10937.096325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10942.147539] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10948.079597] LustreError: 290043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10948.099516] LustreError: 290043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10951.224654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10956.670714] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10965.409390] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10970.318680] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10976.651671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10983.186437] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [10990.031848] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11013.676229] loop1: detected capacity change from 0 to 204800000 [11013.831339] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11013.853297] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11013.871947] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11018.931376] loop1: detected capacity change from 0 to 204800000 [11024.563441] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11024.643854] LustreError: 290033:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs26601: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11024.656125] LustreError: 290033:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs26601 [11024.662665] LustreError: 290033:0:(mgs_handler.c:517:mgs_target_reg()) fs26601-MDT0001: Error checking target index: rc = -13 [11024.672267] LustreError: fs26601-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11024.683963] LustreError: 293781:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11024.698685] LustreError: 293781:0:(tgt_mount.c:2132:server_put_super()) no obd fs26601-MDT0001 [11024.706223] LustreError: 293781:0:(tgt_mount.c:116:server_deregister_mount()) fs26601-MDT0001 not registered [11024.740467] Lustre: server umount fs26601-MDT0001 complete [11024.745665] LustreError: 293781:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11032.930751] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11033.169857] Lustre: Setting parameter fs26601-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs26601-MDT0001 [11033.234743] Lustre: srv-fs26601-MDT0001: No data found on store. Initialize space. [11033.240691] Lustre: Skipped 4 previous similar messages [11033.381943] Lustre: fs26601-MDT0001: new disk, initializing [11033.386457] Lustre: Skipped 3 previous similar messages [11033.516419] Lustre: srv-fs26601-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11033.533156] Lustre: 294247:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26601-MDT0001-mdtlov: can't create new llog: rc = -115 [11033.545732] LustreError: 294247:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26601-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11038.349147] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11044.684335] Lustre: server umount fs26601-MDT0001 complete [11056.348681] loop1: detected capacity change from 0 to 204800000 [11061.617962] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11061.907458] Lustre: 295815:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26601-MDT0001-mdtlov: can't create new llog: rc = -115 [11061.924783] LustreError: 295815:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26601-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11063.269163] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11066.370458] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11070.736062] Lustre: server umount fs26601-MDT0001 complete [11078.636364] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11082.763938] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [11082.771443] Lustre: Skipped 12 previous similar messages [11083.747859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11083.753766] Lustre: Skipped 2 previous similar messages [11086.581231] Lustre: server umount lustre-MDT0000 complete [11094.925672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11095.035704] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11100.660554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11100.964865] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11109.175529] loop1: detected capacity change from 0 to 204800000 [11115.285676] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11115.789253] Lustre: 298258:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs26601-MDT0001-mdtlov: can't create new llog: rc = -115 [11115.792822] LustreError: 298258:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs26601-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11119.543946] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11124.422575] Lustre: server umount fs26601-MDT0001 complete [11146.747181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11146.768673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11146.782688] Lustre: Skipped 3 previous similar messages [11151.685258] Lustre: server umount lustre-OST0000 complete [11156.961623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11161.657835] Lustre: server umount lustre-MDT0000 complete [11164.633600] LustreError: 290025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559315 with bad export cookie 1675319114216878052 [11164.636384] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11164.645072] LustreError: 290025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11164.894856] Lustre: server umount lustre-MDT0001 complete [11173.615777] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [11176.416568] Key type lgssc unregistered [11176.741213] LNet: 300605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11176.751239] LNetError: 300605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11176.777070] LNet: Removed LNI 192.168.204.146@tcp [11177.827732] Key type .llcrypt unregistered [11177.829420] Key type ._llcrypt unregistered [11198.157586] Key type ._llcrypt registered [11198.160984] Key type .llcrypt registered [11198.268695] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11199.331778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11199.364259] alg: No test for adler32 (adler32-zlib) [11200.410776] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [11200.705566] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [11202.367746] Key type lgssc registered [11203.676531] Lustre: Echo OBD driver; http://www.lustre.org/ [11211.179809] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11211.210227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11212.771560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11217.553468] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11223.019480] LustreError: 301876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11223.050495] LustreError: 301876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11226.605375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11226.937635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11232.007493] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11240.976924] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11247.433890] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11254.870674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11255.282300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11257.320625] LustreError: 303707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11261.762101] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11262.445505] LustreError: 303708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11267.557591] LustreError: 303708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11269.594504] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11271.657381] LustreError: 303708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11271.685629] LustreError: 303708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11281.919825] LustreError: 303707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11281.960535] LustreError: 303707:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11292.211396] loop1: detected capacity change from 0 to 204800000 [11292.497813] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11292.505791] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11292.514871] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11298.605929] loop1: detected capacity change from 0 to 204800000 [11302.375901] LustreError: 303709:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11302.388592] LustreError: 303709:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [11303.382429] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11303.523557] Lustre: fs3786-OST0000: new disk, initializing [11303.526966] Lustre: srv-fs3786-OST0000: No data found on store. Initialize space. [11303.531490] Lustre: 305614:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11303.639900] Lustre: fs3786-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11309.805365] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11317.608411] Lustre: server umount fs3786-OST0000 complete [11329.640526] loop1: detected capacity change from 0 to 204800000 [11335.424252] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11335.644408] Lustre: fs3786-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11338.211299] LustreError: 303708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11338.232268] LustreError: 303708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 20 previous similar messages [11338.237149] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11341.277947] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11349.125463] loop2: detected capacity change from 0 to 204800000 [11349.389763] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11349.399579] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11349.411352] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11353.584943] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11354.405433] loop2: detected capacity change from 0 to 204800000 [11360.080324] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11360.199371] LustreError: fs3786-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11360.228110] LustreError: fs3786-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11360.236530] LustreError: 308652:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -13 [11360.253663] LustreError: 308652:0:(tgt_mount.c:2132:server_put_super()) no obd fs3786-OST0001 [11360.261167] LustreError: 308652:0:(tgt_mount.c:116:server_deregister_mount()) fs3786-OST0001 not registered [11360.300451] Lustre: server umount fs3786-OST0001 complete [11360.305308] LustreError: 308652:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11369.690871] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11369.875478] Lustre: fs3786-OST0001: new disk, initializing [11369.881455] Lustre: srv-fs3786-OST0001: No data found on store. Initialize space. [11369.889897] Lustre: 309203:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11369.972698] Lustre: fs3786-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11375.350470] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11384.324951] Lustre: server umount fs3786-OST0000 complete [11394.323964] Lustre: server umount fs3786-OST0001 complete [11404.775072] LustreError: 303847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11404.804491] LustreError: 303847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [11407.206239] loop1: detected capacity change from 0 to 204800000 [11412.017924] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11412.436631] Lustre: fs3786-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11419.161281] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11427.775864] loop2: detected capacity change from 0 to 204800000 [11433.058660] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11440.474124] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11451.442928] Lustre: server umount fs3786-OST0000 complete [11461.653828] Lustre: server umount fs3786-OST0001 complete [11476.461449] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11476.484522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11480.742915] Lustre: server umount lustre-OST0000 complete [11486.689042] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11486.701185] Lustre: Skipped 2 previous similar messages [11486.701426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11486.724862] Lustre: Skipped 2 previous similar messages [11491.333542] Lustre: server umount lustre-MDT0000 complete [11495.163780] LustreError: 306358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559646 with bad export cookie 6389521140072549856 [11495.183866] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11495.759907] Lustre: server umount lustre-MDT0001 complete [11508.589274] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [11513.533160] Key type lgssc unregistered [11514.004551] LNet: 315134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11514.052704] LNetError: 315134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11514.085523] LNet: Removed LNI 192.168.204.146@tcp [11515.938154] Key type .llcrypt unregistered [11515.940698] Key type ._llcrypt unregistered [11557.089641] Key type ._llcrypt registered [11557.091880] Key type .llcrypt registered [11557.185308] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [11574.308993] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11575.363638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11575.461945] alg: No test for adler32 (adler32-zlib) [11576.410740] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [11576.601280] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [11578.281983] Key type lgssc registered [11580.028225] Lustre: Echo OBD driver; http://www.lustre.org/ [11624.099409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11624.145695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11625.436078] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11625.532544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11625.613565] Lustre: lustre-MDT0000: new disk, initializing [11625.712245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11625.728342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11630.947528] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11643.062091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11643.194509] Lustre: 318999:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11643.277436] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11643.283527] Lustre: Skipped 1 previous similar message [11643.373354] Lustre: lustre-MDT0001: new disk, initializing [11643.530110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11643.609846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11643.622724] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11647.735217] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11656.257692] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11661.631736] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11670.559780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11671.128711] Lustre: lustre-OST0000: new disk, initializing [11671.136955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11671.144259] Lustre: 320967:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11671.296640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11677.871920] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11678.858092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11678.873305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11679.094686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11688.372260] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11691.322641] Lustre: Mounted lustre-client - version 2.17.57_83_g6b3cfd8 [11692.921171] Lustre: Unmounted lustre-client [11706.858890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11706.880583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11706.886754] Lustre: Skipped 1 previous similar message [11711.810781] Lustre: server umount lustre-OST0000 complete [11717.088700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11717.091427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11717.119273] Lustre: Skipped 2 previous similar messages [11722.217788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11722.235736] Lustre: Skipped 2 previous similar messages [11723.278028] Lustre: server umount lustre-MDT0000 complete [11729.466281] LustreError: 320966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787559880 with bad export cookie 13330032071129229984 [11729.500049] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11729.501114] LustreError: 320966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11730.064797] Lustre: server umount lustre-MDT0001 complete [11742.980701] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [11746.370031] Key type lgssc unregistered [11746.656292] LNet: 1 local NIs in recovery (showing 1): 192.168.204.146@tcp [11746.742910] LNet: 323136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11746.752019] LNetError: 323136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11822.577322] LNet: Removed LNI 192.168.204.146@tcp [11823.829696] Key type .llcrypt unregistered [11823.835158] Key type ._llcrypt unregistered [11847.552859] Key type ._llcrypt registered [11847.556683] Key type .llcrypt registered [11847.717440] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [11867.389022] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [11868.552262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11868.644796] alg: No test for adler32 (adler32-zlib) [11869.645914] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [11870.044826] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [11871.687204] Key type lgssc registered [11873.219930] Lustre: Echo OBD driver; http://www.lustre.org/ [11911.206292] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 04:27:40 (1787560060) [11917.530299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11917.569977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11918.798994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11918.853177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11918.915890] Lustre: lustre-MDT0000: new disk, initializing [11919.027779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11919.057435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11924.063819] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11939.487049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11939.639930] Lustre: 326358:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11939.738510] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11939.751934] Lustre: Skipped 1 previous similar message [11939.942475] Lustre: lustre-MDT0001: new disk, initializing [11940.163901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11940.220913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11940.258425] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11946.551976] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11959.351989] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11969.010022] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11979.097905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11979.430324] Lustre: lustre-OST0000: new disk, initializing [11979.433379] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11979.447207] Lustre: 328333:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11979.549807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11988.357947] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [11988.668528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11988.679132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11988.889173] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12002.123216] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12024.801626] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12024.805189] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12024.826542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12024.839983] Lustre: Skipped 1 previous similar message [12028.794620] Lustre: server umount lustre-OST0000 complete [12035.043469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12035.064867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12035.086822] Lustre: Skipped 2 previous similar messages [12040.160909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12040.166511] Lustre: Skipped 2 previous similar messages [12044.261262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12044.270681] Lustre: Skipped 1 previous similar message [12047.327083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12047.526923] Lustre: server umount lustre-MDT0000 complete [12050.882195] LustreError: 327167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560201 with bad export cookie 3002790735500950303 [12050.897772] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12050.900067] LustreError: 327167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12051.224173] Lustre: server umount lustre-MDT0001 complete [12065.399756] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12068.511405] Key type lgssc unregistered [12068.983689] LNet: 330245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12068.994405] LNetError: 330245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12070.053990] LNet: Removed LNI 192.168.204.146@tcp [12071.803480] Key type .llcrypt unregistered [12071.813102] Key type ._llcrypt unregistered [12098.632850] Key type ._llcrypt registered [12098.638201] Key type .llcrypt registered [12098.780186] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12099.826996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12099.988669] alg: No test for adler32 (adler32-zlib) [12101.057548] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [12101.460382] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12103.137756] Key type lgssc registered [12104.759939] Lustre: Echo OBD driver; http://www.lustre.org/ [12115.335495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12115.411689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12117.180833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12123.621238] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12127.723040] LustreError: 331522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12127.772601] LustreError: 331522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12132.851617] LustreError: 331523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12135.496589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12136.535155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12144.584914] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12153.715812] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12159.701285] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12167.859424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12168.258269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12176.293837] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12185.454974] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12199.594398] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12201.547757] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12206.607469] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [12208.098921] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12208.107125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12208.107744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12208.120410] Lustre: Skipped 2 previous similar messages [12208.147356] Lustre: Skipped 2 previous similar messages [12211.702086] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.46@tcp (stopping) [12212.373496] Lustre: server umount lustre-MDT0000 complete [12213.232088] LustreError: 331517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12213.262810] LustreError: 331517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12216.523575] LustreError: 331503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560367 with bad export cookie 4560761240219145681 [12216.525401] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12216.539524] LustreError: 331503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12216.818941] LustreError: 331518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.46@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12216.874969] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.46@tcp (stopping) [12218.342359] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12221.344890] LustreError: 331519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12221.351402] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12221.376649] LustreError: 331519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12221.380131] Lustre: Skipped 1 previous similar message [12222.998329] Lustre: server umount lustre-MDT0001 complete [12226.915585] Lustre: server umount lustre-OST0000 complete [12237.735765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12242.567455] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12245.213209] Lustre: 335705:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.146@tcp [12246.933179] Lustre: server umount MGS complete [12257.679408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12258.191955] LustreError: 336278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12258.229803] LustreError: 336278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12258.399221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12263.687660] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12276.031773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12276.551457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12281.403801] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12289.648381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12290.183286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12298.038212] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12306.460505] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12307.995941] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12315.104464] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12316.787919] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12322.273885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12322.290892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12322.308354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12322.323377] Lustre: Skipped 1 previous similar message [12323.483957] Lustre: server umount lustre-OST0000 complete [12329.446843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12329.483898] Lustre: Skipped 2 previous similar messages [12334.564765] LustreError: 336279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12334.597589] LustreError: 336279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12334.908591] Lustre: server umount lustre-MDT0000 complete [12338.895947] LustreError: 336260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560489 with bad export cookie 4560761240219146521 [12338.910884] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12338.916126] LustreError: 336260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12339.284039] Lustre: server umount lustre-MDT0001 complete [12351.863267] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12355.567760] Key type lgssc unregistered [12355.839243] LNet: 339717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12355.848079] LNetError: 339717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12355.868298] LNet: Removed LNI 192.168.204.146@tcp [12357.314908] Key type .llcrypt unregistered [12357.317072] Key type ._llcrypt unregistered [12378.422724] Key type ._llcrypt registered [12378.425691] Key type .llcrypt registered [12378.601582] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [12396.443396] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12397.622037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12397.666317] alg: No test for adler32 (adler32-zlib) [12398.896369] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [12399.172936] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [12400.847519] Key type lgssc registered [12402.044688] Lustre: Echo OBD driver; http://www.lustre.org/ [12440.554434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12440.582900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12441.819939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12441.851281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12441.979516] Lustre: lustre-MDT0000: new disk, initializing [12442.069751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12442.084265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12447.104837] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12459.265860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12459.442069] Lustre: 342878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12459.550490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12459.554287] Lustre: Skipped 1 previous similar message [12459.670447] Lustre: lustre-MDT0001: new disk, initializing [12459.790017] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12459.838284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12459.849717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12464.954267] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12475.809866] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12481.596760] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12489.019815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12489.334229] Lustre: lustre-OST0000: new disk, initializing [12489.337377] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12489.345374] Lustre: 344848:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12489.437841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12494.930106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12494.946307] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12495.090975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12495.694287] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12505.498566] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12510.879091] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12511.140750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12516.184848] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12516.395543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12520.950823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12520.960613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12520.973118] LustreError: Skipped 1 previous similar message [12520.974280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12521.009272] Lustre: Skipped 1 previous similar message [12524.858798] Lustre: server umount lustre-OST0000 complete [12531.175921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12531.197974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12531.208643] Lustre: Skipped 2 previous similar messages [12535.714354] Lustre: server umount lustre-MDT0000 complete [12540.031797] LustreError: 342872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560690 with bad export cookie 8772093801581116124 [12540.033909] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12540.050712] LustreError: 342872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12540.343160] Lustre: server umount lustre-MDT0001 complete [12549.745980] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 04:38:19 (1787560699) [12571.662405] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [12581.794631] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [12622.490873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12622.644706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12622.678120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12622.728465] Lustre: lustre-MDT0000: new disk, initializing [12622.816627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12622.838979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12627.481840] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12637.748117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12637.943613] Lustre: 349609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12638.044241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12638.053830] Lustre: Skipped 1 previous similar message [12638.150321] Lustre: lustre-MDT0001: new disk, initializing [12638.290100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12638.327053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12638.340289] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12643.756814] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12654.413773] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12660.227343] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12667.095368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12667.355147] Lustre: lustre-OST0000: new disk, initializing [12667.361196] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12667.372507] Lustre: 351574:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12667.503898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12673.495823] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [12675.617400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12675.625839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12675.685435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12681.622386] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12726.757756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12726.758730] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12726.776777] LustreError: Skipped 1 previous similar message [12726.780131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12726.798726] Lustre: Skipped 2 previous similar messages [12726.828586] Lustre: Skipped 1 previous similar message [12728.664210] Lustre: server umount lustre-OST0000 complete [12735.458831] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12735.471150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12735.486592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12739.536903] Lustre: server umount lustre-MDT0000 complete [12742.117243] LustreError: 349621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12742.160083] LustreError: 349621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12743.543701] LustreError: 352445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560894 with bad export cookie 8772093801581117006 [12743.555511] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12743.847025] Lustre: server umount lustre-MDT0001 complete [12755.052591] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12758.901254] Key type lgssc unregistered [12759.350043] LNet: 353756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12759.357292] LNetError: 353756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12759.378856] LNet: Removed LNI 192.168.204.146@tcp [12760.788246] Key type .llcrypt unregistered [12760.794788] Key type ._llcrypt unregistered [12859.309235] Key type ._llcrypt registered [12859.314899] Key type .llcrypt registered [12859.469824] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12859.701528] Key type .llcrypt unregistered [12859.705433] Key type ._llcrypt unregistered [12944.784246] Key type ._llcrypt registered [12944.790442] Key type .llcrypt registered [12944.972821] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12945.405714] Key type .llcrypt unregistered [12945.418672] Key type ._llcrypt unregistered [12979.220227] Key type ._llcrypt registered [12979.224381] Key type .llcrypt registered [12979.339202] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [12979.661350] Key type .llcrypt unregistered [12979.665098] Key type ._llcrypt unregistered [13011.563644] Key type ._llcrypt registered [13011.568311] Key type .llcrypt registered [13011.740962] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13012.143797] Key type .llcrypt unregistered [13012.146043] Key type ._llcrypt unregistered [13094.675841] Key type ._llcrypt registered [13094.677310] Key type .llcrypt registered [13094.846407] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13095.322549] Key type .llcrypt unregistered [13095.325141] Key type ._llcrypt unregistered [13127.253834] Key type ._llcrypt registered [13127.256615] Key type .llcrypt registered [13127.384801] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13127.730531] Key type .llcrypt unregistered [13127.733733] Key type ._llcrypt unregistered [13159.270828] Key type ._llcrypt registered [13159.278727] Key type .llcrypt registered [13159.557390] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13160.352278] Key type .llcrypt unregistered [13160.353933] Key type ._llcrypt unregistered [13198.453850] Key type ._llcrypt registered [13198.462523] Key type .llcrypt registered [13198.756487] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13199.068392] Key type .llcrypt unregistered [13199.079353] Key type ._llcrypt unregistered [13231.313192] Key type ._llcrypt registered [13231.314882] Key type .llcrypt registered [13231.484526] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13232.039370] Key type .llcrypt unregistered [13232.042743] Key type ._llcrypt unregistered [13265.562307] Key type ._llcrypt registered [13265.564354] Key type .llcrypt registered [13265.649738] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13265.888607] Key type .llcrypt unregistered [13265.892493] Key type ._llcrypt unregistered [13329.260853] Key type ._llcrypt registered [13329.262987] Key type .llcrypt registered [13329.475529] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13330.179149] Key type .llcrypt unregistered [13330.185669] Key type ._llcrypt unregistered [13363.401621] Key type ._llcrypt registered [13363.407034] Key type .llcrypt registered [13363.506244] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13363.816358] Key type .llcrypt unregistered [13363.818595] Key type ._llcrypt unregistered [13393.917447] Key type ._llcrypt registered [13393.919667] Key type .llcrypt registered [13394.022801] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13394.314735] Key type .llcrypt unregistered [13394.318788] Key type ._llcrypt unregistered [13449.818145] Key type ._llcrypt registered [13449.824332] Key type .llcrypt registered [13449.944753] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13466.248286] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13467.756768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13467.922148] alg: No test for adler32 (adler32-zlib) [13469.229592] Lustre: Lustre: Build Version: 2.17.57_83_g6b3cfd8 [13469.555312] LNet: Added LNI 192.168.204.146@tcp [8/256/0/180] [13471.223540] Key type lgssc registered [13472.874525] Lustre: Echo OBD driver; http://www.lustre.org/ [13511.559900] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13511.594606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13512.878195] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13512.899925] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13513.004469] Lustre: tst250fs-MDT0000: new disk, initializing [13513.147121] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13513.176232] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13518.388239] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13529.587640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13529.725912] Lustre: 374109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13529.786461] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13529.790975] Lustre: Skipped 1 previous similar message [13529.903183] Lustre: tst250fs-MDT0001: new disk, initializing [13529.984125] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13530.041632] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13530.053097] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13535.110393] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13539.888717] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13546.903308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13547.156496] Lustre: tst250fs-OST0000: new disk, initializing [13547.159837] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13547.165568] Lustre: 375741:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13547.263585] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13553.812356] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13556.311769] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13556.329752] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13556.419540] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13564.279364] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13564.568615] Lustre: tst250fs-OST0001: new disk, initializing [13564.579528] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13564.588969] Lustre: 376612:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13564.676829] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13564.905995] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13564.925606] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13565.008813] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13571.382747] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_default_debug -1 all [13577.567019] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13580.578808] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13580.584199] Lustre: Skipped 2 previous similar messages [13600.747320] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13600.751457] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13600.771845] Lustre: Skipped 2 previous similar messages [13600.783758] Lustre: Skipped 3 previous similar messages [13605.869505] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13605.881169] Lustre: Skipped 3 previous similar messages [13606.021648] Lustre: server umount tst250fs-MDT0000 complete [13609.471155] LustreError: 374102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561760 with bad export cookie 16718289519551008974 [13609.473180] LustreError: MGC192.168.204.146@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13609.479275] LustreError: 374102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13610.012339] Lustre: server umount tst250fs-MDT0001 complete [13624.751381] Lustre: server umount tst250fs-OST0000 complete [13626.338798] Lustre: 372334:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787561761/real 1787561761] req@ffff93b705bbca80 x1874394213320320/t0(0) o400->tst250fs-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787561777 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13626.372660] Lustre: tst250fs-MDT0001-lwp-OST0001: Connection to tst250fs-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13626.385899] Lustre: Skipped 1 previous similar message [13628.825843] Lustre: server umount tst250fs-OST0001 complete [13648.317359] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13656.434766] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13687.853924] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 04:57:17 (1787561837) [13689.520257] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13707.227784] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing set_hostid [13717.417068] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing load_modules_local [13748.000913] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13397 sec ========= 04:58:18 (1787561898) [13749.642098] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:58:19 (1787561899) === [13752.887275] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:58:23 (1787561903) === [13790.598952] Lustre: DEBUG MARKER: oleg446-server.virtnet: executing unload_modules_local [13793.051352] Key type lgssc unregistered [13793.354498] LNet: 385744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13793.367455] LNetError: 385744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13793.385237] LNet: Removed LNI 192.168.204.146@tcp [13794.694220] Key type .llcrypt unregistered [13794.695768] Key type ._llcrypt unregistered