[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 514888776 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002339] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.008007] pid_max: default: 32768 minimum: 301 [ 0.009128] LSM: Security Framework initializing [ 0.010033] Yama: becoming mindful. [ 0.012023] SELinux: Initializing. [ 0.013044] *** VALIDATE selinux *** [ 0.020410] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024694] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025120] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026081] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027082] *** VALIDATE tmpfs *** [ 0.028406] *** VALIDATE proc *** [ 0.029193] *** VALIDATE cgroup *** [ 0.030005] *** VALIDATE cgroup2 *** [ 0.032084] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033126] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035023] Spectre V2 : User space: Vulnerable [ 0.036007] Speculative Store Bypass: Vulnerable [ 0.039227] debug: unmapping init [mem 0xffffffff89a59000-0xffffffff89a60fff] [ 0.041249] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042642] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043019] ... version: 2 [ 0.044009] ... bit width: 48 [ 0.045008] ... generic registers: 4 [ 0.046007] ... value mask: 0000ffffffffffff [ 0.047009] ... max period: 00007fffffffffff [ 0.048008] ... fixed-purpose events: 3 [ 0.049006] ... event mask: 000000070000000f [ 0.051184] rcu: Hierarchical SRCU implementation. [ 0.053358] smp: Bringing up secondary CPUs ... [ 0.054498] x86: Booting SMP configuration: [ 0.055020] .... node #0, CPUs: #1 #2 #3 [ 0.062543] smp: Brought up 1 node, 4 CPUs [ 0.064009] smpboot: Max logical packages: 1 [ 0.065009] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.124497] node 0 deferred pages initialised in 57ms [ 0.128011] devtmpfs: initialized [ 0.129171] x86/mm: Memory block size: 128MB [ 0.131737] gcov: version magic: 0x41383552 [ 0.133324] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.134057] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.135207] pinctrl core: initialized pinctrl subsystem [ 0.136157] [ 0.136630] ************************************************************* [ 0.137009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.138008] ** ** [ 0.139007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.140007] ** ** [ 0.141008] ** This means that this kernel is built to expose internal ** [ 0.142007] ** IOMMU data structures, which may compromise security on ** [ 0.143009] ** your system. ** [ 0.144008] ** ** [ 0.145008] ** If you see this message and you are not debugging the ** [ 0.146008] ** kernel, report this immediately to your vendor! ** [ 0.147007] ** ** [ 0.148008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149008] ************************************************************* [ 0.150722] NET: Registered protocol family 16 [ 0.151403] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.152046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.153057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.154452] cpuidle: using governor menu [ 0.156577] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.159522] PCI: Using configuration type 1 for base access [ 0.161182] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.172331] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.175054] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.180033] cryptd: max_cpu_qlen set to 1000 [ 0.183243] ACPI: Added _OSI(Module Device) [ 0.185009] ACPI: Added _OSI(Processor Device) [ 0.187013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.189010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.194485] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.201035] ACPI: Interpreter enabled [ 0.203049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.204009] ACPI: Using IOAPIC for interrupt routing [ 0.205080] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.208340] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.221596] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.223027] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.226015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.229067] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.234473] acpiphp: Slot [2] registered [ 0.236164] acpiphp: Slot [5] registered [ 0.238164] acpiphp: Slot [6] registered [ 0.239133] acpiphp: Slot [7] registered [ 0.241104] acpiphp: Slot [8] registered [ 0.242136] acpiphp: Slot [9] registered [ 0.244128] acpiphp: Slot [10] registered [ 0.245106] acpiphp: Slot [3] registered [ 0.246074] acpiphp: Slot [4] registered [ 0.248118] acpiphp: Slot [11] registered [ 0.250080] acpiphp: Slot [12] registered [ 0.251072] acpiphp: Slot [13] registered [ 0.252067] acpiphp: Slot [14] registered [ 0.253108] acpiphp: Slot [15] registered [ 0.255062] acpiphp: Slot [16] registered [ 0.256081] acpiphp: Slot [17] registered [ 0.257064] acpiphp: Slot [18] registered [ 0.258060] acpiphp: Slot [19] registered [ 0.259117] acpiphp: Slot [20] registered [ 0.261062] acpiphp: Slot [21] registered [ 0.262110] acpiphp: Slot [22] registered [ 0.263077] acpiphp: Slot [23] registered [ 0.264060] acpiphp: Slot [24] registered [ 0.266063] acpiphp: Slot [25] registered [ 0.267063] acpiphp: Slot [26] registered [ 0.268056] acpiphp: Slot [27] registered [ 0.269057] acpiphp: Slot [28] registered [ 0.270057] acpiphp: Slot [29] registered [ 0.271079] acpiphp: Slot [30] registered [ 0.272138] acpiphp: Slot [31] registered [ 0.274068] PCI host bridge to bus 0000:00 [ 0.275012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.277024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.279017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.281012] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.283047] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.286017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.288152] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.290995] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.294176] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.303845] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.309052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.312016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.314021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.317014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.320319] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.322789] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.325032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.327748] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.334012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.346013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.351012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.357000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.378015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.403016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.440016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.457881] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.469015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.478012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.495012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.505334] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.513015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.520017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.539014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.552682] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.575014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.591013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.618014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.635575] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.648014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.662021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.690017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.708229] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.723015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.735021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.782019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.809402] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.812346] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.814255] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.816321] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.818179] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.823034] iommu: Default domain type: Passthrough [ 0.824378] SCSI subsystem initialized [ 0.826100] ACPI: bus type USB registered [ 0.827085] usbcore: registered new interface driver usbfs [ 0.829085] usbcore: registered new interface driver hub [ 0.830105] usbcore: registered new device driver usb [ 0.832198] pps_core: LinuxPPS API ver. 1 registered [ 0.833005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.836046] PTP clock support registered [ 0.838046] EDAC MC: Ver: 3.0.0 [ 0.839409] PCI: Using ACPI for IRQ routing [ 0.841426] NetLabel: Initializing [ 0.842008] NetLabel: domain hash size = 128 [ 0.843006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.845058] NetLabel: unlabeled traffic allowed by default [ 0.847032] vgaarb: loaded [ 0.848250] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.849009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.856208] clocksource: Switched to clocksource kvm-clock [ 0.967159] VFS: Disk quotas dquot_6.6.0 [ 0.968406] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.970332] *** VALIDATE ramfs *** [ 0.971290] *** VALIDATE hugetlbfs *** [ 0.972510] pnp: PnP ACPI init [ 0.974562] pnp: PnP ACPI: found 6 devices [ 0.989787] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.992616] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.994233] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.996074] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.997830] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.999771] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.002096] NET: Registered protocol family 2 [ 1.003738] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.007862] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.010858] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.015773] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.018882] TCP: Hash tables configured (established 65536 bind 65536) [ 1.021531] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.024072] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.026449] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.028876] NET: Registered protocol family 1 [ 1.032211] RPC: Registered named UNIX socket transport module. [ 1.034139] RPC: Registered udp transport module. [ 1.035544] RPC: Registered tcp transport module. [ 1.036879] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.039820] NET: Registered protocol family 44 [ 1.041177] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.042854] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.044449] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.046243] PCI: CLS 0 bytes, default 64 [ 1.048129] Unpacking initramfs... [ 2.477320] debug: unmapping init [mem 0xffff8ccfbcc54000-0xffff8ccfbffbffff] [ 2.480862] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.482595] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.484935] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 2.990468] Initialise system trusted keyrings [ 2.992241] Key type blacklist registered [ 2.994690] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.002711] zbud: loaded [ 3.005126] *** VALIDATE nfs *** [ 3.006085] *** VALIDATE nfs4 *** [ 3.007180] pstore: using deflate compression [ 3.010440] Platform Keyring initialized [ 3.114268] NET: Registered protocol family 38 [ 3.115767] Key type asymmetric registered [ 3.117482] Asymmetric key parser 'x509' registered [ 3.119519] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.122493] io scheduler mq-deadline registered [ 3.124434] io scheduler kyber registered [ 3.126086] io scheduler bfq registered [ 3.127912] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.131115] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.133127] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.135263] ACPI: Power Button [PWRF] [ 3.140914] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.148028] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.164773] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.178784] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.202044] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.232226] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.261462] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.266582] Non-volatile memory driver v1.3 [ 3.267823] Linux agpgart interface v0.103 [ 3.305305] virtio_blk virtio1: [vda] 145160 512-byte logical blocks (74.3 MB/70.9 MiB) [ 3.307648] vda: detected capacity change from 0 to 74321920 [ 3.323097] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.326584] vdb: detected capacity change from 0 to 1073741824 [ 3.340915] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.343382] vdc: detected capacity change from 0 to 2621440000 [ 3.357076] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.359758] vdd: detected capacity change from 0 to 2621440000 [ 3.373700] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.376198] vde: detected capacity change from 0 to 4294967296 [ 3.396327] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.398825] vdf: detected capacity change from 0 to 4294967296 [ 3.408847] libphy: Fixed MDIO Bus: probed [ 3.416269] usbcore: registered new interface driver usbserial_generic [ 3.418238] usbserial: USB Serial support registered for generic [ 3.420396] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.425521] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.426851] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.428711] mousedev: PS/2 mouse device common for all mice [ 3.431196] rtc_cmos 00:05: RTC can wake from S4 [ 3.434042] rtc_cmos 00:05: registered as rtc0 [ 3.435511] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.439579] intel_pstate: CPU model not supported [ 3.441538] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.445238] hid: raw HID events driver (C) Jiri Kosina [ 3.449044] usbcore: registered new interface driver usbhid [ 3.451406] usbhid: USB HID core driver [ 3.452821] drop_monitor: Initializing network drop monitor service [ 3.455357] Initializing XFRM netlink socket [ 3.457213] NET: Registered protocol family 10 [ 3.460504] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.460508] Segment Routing with IPv6 [ 3.460550] NET: Registered protocol family 17 [ 3.466754] mpls_gso: MPLS GSO support [ 3.472114] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.473047] RAS: Correctable Errors collector initialized. [ 3.477077] AVX version of gcm_enc/dec engaged. [ 3.478677] AES CTR mode by8 optimization enabled [ 3.544978] sched_clock: Marking stable (3544899670, 0)->(4448052951, -903153281) [ 3.548448] registered taskstats version 1 [ 3.550377] Loading compiled-in X.509 certificates [ 3.551873] zswap: loaded using pool lzo/zbud [ 3.573677] Key type big_key registered [ 3.584474] Key type encrypted registered [ 3.585992] ima: No TPM chip found, activating TPM-bypass! [ 3.588124] ima: Allocated hash algorithm: sha1 [ 3.589873] ima: No architecture policies found [ 3.591675] evm: Initialising EVM extended attributes: [ 3.593642] evm: security.selinux [ 3.594851] evm: security.ima [ 3.596091] evm: security.capability [ 3.597564] evm: HMAC attrs: 0x1 [ 3.600107] rtc_cmos 00:05: setting system clock to 2026-06-25 04:15:41 UTC (1782360941) [ 3.606899] debug: unmapping init [mem 0xffffffff8aa03000-0xffffffff8abfffff] [ 3.610201] debug: unmapping init [mem 0xffffffff89782000-0xffffffff89a58fff] [ 3.618116] Write protecting the kernel read-only data: 28672k [ 3.622594] debug: unmapping init [mem 0xffffffff87e03000-0xffffffff87ffffff] [ 3.625578] debug: unmapping init [mem 0xffffffff88714000-0xffffffff887fffff] [ 3.667236] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.676855] systemd[1]: Detected virtualization kvm. [ 3.678898] systemd[1]: Detected architecture x86-64. [ 3.680807] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.703630] systemd[1]: No hostname configured. [ 3.705292] systemd[1]: Set hostname to . [ 3.707268] random: systemd: uninitialized urandom read (16 bytes read) [ 3.709593] systemd[1]: Initializing machine ID from random generator. [ 3.765891] random: ln: uninitialized urandom read (6 bytes read) [ 3.849607] random: systemd: uninitialized urandom read (16 bytes read) [ 3.855931] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.860684] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.867597] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.623675] device-mapper: uevent: version 1.0.3 [ 4.625894] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.741679] virtio_net virtio0 ens2: renamed from eth0 [ 5.893206] random: fast init done [ 6.118500] scsi host0: ata_piix [ 6.256728] scsi host1: ata_piix [ 6.259573] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.270709] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.104564] random: crng init done [ 11.105796] random: 7 urandom warning(s) missed due to ratelimiting [ 11.159717] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.422542] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.877678] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.158256] SELinux: Disabled at runtime. [ 14.216672] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.225710] systemd[1]: Detected virtualization kvm. [ 14.227495] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.819599] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.823253] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.829311] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.834036] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.837850] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.847758] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.856898] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ 15.096908] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt[ 15.409346] squashfs: version 4.0 (2009/01/31) Phillip Lougher . [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.871794] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.888382] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.346275] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.355427] EDAC sbridge: Ver: 1.1.2 [ 18.348405] Key type dns_resolver registered [ 18.715959] NFS: Registering the id_resolver key type [ 18.718075] Key type id_resolver registered [ 18.719632] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg204-server login: [ 67.571713] libcfs: loading out-of-tree module taints kernel. [ 67.619528] Key type ._llcrypt registered [ 67.621215] Key type .llcrypt registered [ 67.685820] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 79.759402] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 80.478345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 80.484242] alg: No test for adler32 (adler32-zlib) [ 81.535372] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 81.910653] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 83.559249] Key type lgssc registered [ 84.221107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 91.933993] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 110.404949] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 116.834453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 116.845242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 117.966370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 117.984072] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 118.032975] Lustre: lustre-MDT0000: new disk, initializing [ 118.088268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 118.094754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 119.809933] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 126.324616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 126.368696] Lustre: 6513:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 126.387715] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 126.390504] Lustre: Skipped 1 previous similar message [ 126.432984] Lustre: lustre-MDT0001: new disk, initializing [ 126.462422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 126.478754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 126.486937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 128.254266] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 130.938640] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 134.946030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 135.049269] Lustre: lustre-OST0000: new disk, initializing [ 135.051421] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 135.053546] Lustre: 8452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 135.076482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 137.313406] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 140.790616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 140.794570] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 140.812876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 143.743665] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 143.809265] Lustre: lustre-OST0001: new disk, initializing [ 143.811728] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 143.815365] Lustre: 9524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 143.844485] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 146.258792] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 150.519229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 150.523276] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 150.550276] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 153.016249] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 158.966974] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 161.573814] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing check_logdir /tmp/testlogs/ [ 163.479478] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing yml_node [ 165.476929] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 166.849671] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 168.247988] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 169.116310] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 25 00:18:25 EDT 2026 [ 176.863729] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 177.560703] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 191.455731] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 191.456901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 191.463478] Lustre: Skipped 2 previous similar messages [ 191.465886] Lustre: Skipped 1 previous similar message [ 195.563897] Lustre: server umount lustre-MDT0000 complete [ 197.073406] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361134 with bad export cookie 10868729197590291116 [ 197.074154] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 197.078312] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 201.697235] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 201.697527] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 201.707571] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 201.717745] Lustre: Skipped 1 previous similar message [ 201.721773] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 201.728725] Lustre: Skipped 3 previous similar messages [ 202.719847] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 202.719857] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 202.730132] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 203.243547] Lustre: server umount lustre-MDT0001 complete [ 204.985832] Lustre: server umount lustre-OST0000 complete [ 206.689603] Lustre: server umount lustre-OST0001 complete [ 209.251318] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 212.939814] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 228.694441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 228.799024] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 228.811118] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 228.857955] Lustre: lustre-MDT0000: new disk, initializing [ 228.889917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 228.898071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 230.814938] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 236.097413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 236.156731] Lustre: 15063:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 236.301835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 238.303715] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 242.985345] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 245.546411] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 248.395876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 248.525591] Lustre: lustre-OST0000: new disk, initializing [ 248.527575] Lustre: Skipped 1 previous similar message [ 248.529688] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 248.532037] Lustre: Skipped 2 previous similar messages [ 248.534747] Lustre: 17031:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 248.575251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 248.579235] Lustre: Skipped 1 previous similar message [ 250.252850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 250.257489] Lustre: Skipped 1 previous similar message [ 250.260907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 250.307082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 251.124865] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 255.907762] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 258.228956] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 258.336241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 260.319679] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 260.408602] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 265.698437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 265.698813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 265.699644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 265.699648] Lustre: Skipped 1 previous similar message [ 265.702832] Lustre: Skipped 2 previous similar messages [ 265.706773] LustreError: Skipped 1 previous similar message [ 267.346803] Lustre: server umount lustre-OST0000 complete [ 270.816944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 270.817839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 270.820867] Lustre: Skipped 1 previous similar message [ 270.822598] Lustre: Skipped 1 previous similar message [ 281.059882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 281.062757] Lustre: Skipped 4 previous similar messages [ 283.103221] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 283.192124] Lustre: server umount lustre-MDT0000 complete [ 284.923600] LustreError: 15054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361222 with bad export cookie 10868729197590292649 [ 284.925370] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 284.927585] LustreError: 15054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 285.083090] Lustre: server umount lustre-MDT0001 complete [ 287.666803] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 00:20:24 (1782361224) [ 296.299946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 296.536904] LustreError: 19905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 296.576521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 298.415355] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 302.049479] LustreError: 19906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 302.577561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 304.652266] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 307.891253] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 310.168225] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 311.053496] Lustre: Failing over lustre-MDT0000 [ 311.159784] Lustre: server umount lustre-MDT0000 complete [ 312.288298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.289382] LustreError: 19901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.293433] Lustre: Skipped 1 previous similar message [ 323.554441] LustreError: 19906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 323.565726] LustreError: 19906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 325.378173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 325.439889] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 327.514302] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 328.674110] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 328.676567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 328.693383] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 332.666302] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 335.086953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 335.088837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.099580] Lustre: Skipped 1 previous similar message [ 341.303100] Lustre: server umount lustre-MDT0000 complete [ 343.142624] LustreError: 22560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361281 with bad export cookie 10868729197590294063 [ 343.145052] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 343.151171] LustreError: 22560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 343.285746] Lustre: server umount lustre-MDT0001 complete [ 347.383695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.674854] LustreError: 23237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 347.708036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 347.711930] Lustre: Skipped 2 previous similar messages [ 349.867153] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 354.346101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 356.663223] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 358.768007] hrtimer: interrupt took 6880924 ns [ 360.428396] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 363.500255] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 364.519962] Lustre: Failing over lustre-MDT0000 [ 364.528488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 364.533840] Lustre: Skipped 1 previous similar message [ 364.646930] Lustre: server umount lustre-MDT0000 complete [ 378.953811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 379.028209] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 380.205861] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 380.209590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 380.224497] Lustre: Skipped 1 previous similar message [ 380.232967] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 381.364431] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 386.572494] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 390.624692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 390.625981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 390.635350] Lustre: Skipped 6 previous similar messages [ 394.908495] Lustre: server umount lustre-MDT0000 complete [ 395.746188] LustreError: 23237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 395.758068] LustreError: 23237:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 396.578665] LustreError: 23218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361334 with bad export cookie 10868729197590295001 [ 396.581592] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 396.586404] LustreError: 23218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 396.743364] Lustre: server umount lustre-MDT0001 complete [ 400.847330] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 00:22:17 (1782361337) [ 407.545785] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 412.004103] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 429.150028] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 433.835643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.962127] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 433.978651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 434.018472] Lustre: lustre-MDT0000: new disk, initializing [ 434.066687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 434.071761] Lustre: Skipped 2 previous similar messages [ 434.078871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 435.910780] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 441.014585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.063454] Lustre: 29599:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 441.178993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 442.897227] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 447.592660] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 450.064227] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 453.145553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 453.266790] Lustre: 31566:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 454.938902] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 454.965574] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 454.971839] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 455.899644] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 460.398136] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 463.532572] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 475.617854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 475.619225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.623409] LustreError: Skipped 1 previous similar message [ 475.624141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 475.629312] Lustre: Skipped 3 previous similar messages [ 476.970995] Lustre: server umount lustre-OST0000 complete [ 484.644643] Lustre: server umount lustre-MDT0000 complete [ 485.856548] LustreError: 29605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.863924] LustreError: 29605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 486.068646] LustreError: 30392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361423 with bad export cookie 10868729197590295407 [ 486.071220] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 486.074640] LustreError: 30392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 486.251483] Lustre: server umount lustre-MDT0001 complete [ 490.564209] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 491.880219] Key type lgssc unregistered [ 492.032644] LNet: 33517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 492.037951] LNetError: 33517:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 492.052501] LNet: Removed LNI 192.168.202.104@tcp [ 492.469253] Key type .llcrypt unregistered [ 492.470639] Key type ._llcrypt unregistered [ 503.118868] Key type ._llcrypt registered [ 503.120405] Key type .llcrypt registered [ 503.168526] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 503.633471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 503.645103] alg: No test for adler32 (adler32-zlib) [ 504.526667] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 504.647441] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 506.239159] Key type lgssc registered [ 506.775676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 510.364870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 510.374815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.557737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 513.271256] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 516.877402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 517.055128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 518.652550] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 521.822405] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 524.053536] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 526.898605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 527.048184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.631299] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 532.771360] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 541.421842] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 548.749689] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 00:24:45 (1782361485) [ 549.485478] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 550.341795] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 553.544966] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 00:24:50 (1782361490) [ 557.288132] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 00:24:53 (1782361493) [ 557.833378] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 561.483937] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 561.486883] Lustre: Skipped 2 previous similar messages [ 562.239936] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 00:24:58 (1782361498) [ 565.947250] Lustre: Setting parameter lustre-OST0000-osc.osc.max_pages_per_rpc=1024 in log lustre-client [ 565.950052] Lustre: Skipped 4 previous similar messages [ 572.748071] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 00:25:09 (1782361509) [ 574.084337] Lustre: *** cfs_fail_loc=131b, val=2*** [ 575.698405] Lustre: *** cfs_fail_loc=131b, val=2*** [ 579.072800] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 00:25:15 (1782361515) [ 579.810471] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 579.813016] Lustre: Skipped 9 previous similar messages [ 584.858294] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 00:25:21 (1782361521) [ 592.866343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.867573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 592.868090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 592.871585] Lustre: Skipped 1 previous similar message [ 595.672507] Lustre: server umount lustre-OST0000 complete [ 597.987953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.989141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 597.995911] Lustre: Skipped 1 previous similar message [ 598.001928] Lustre: Skipped 2 previous similar messages [ 603.105902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 603.108537] Lustre: Skipped 1 previous similar message [ 603.552077] Lustre: server umount lustre-MDT0000 complete [ 605.281382] LustreError: 34887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361543 with bad export cookie 13472178672687998114 [ 605.284843] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 605.290867] LustreError: 34887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 605.431290] Lustre: server umount lustre-MDT0001 complete [ 610.110036] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 611.638421] Key type lgssc unregistered [ 611.834675] LNet: 42757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.839091] LNetError: 42757:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 611.848544] LNet: Removed LNI 192.168.202.104@tcp [ 612.268537] Key type .llcrypt unregistered [ 612.269783] Key type ._llcrypt unregistered [ 620.774626] Key type ._llcrypt registered [ 620.776286] Key type .llcrypt registered [ 620.821864] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 629.162936] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 629.700295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 629.713454] alg: No test for adler32 (adler32-zlib) [ 630.644763] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 630.762527] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 632.367215] Key type lgssc registered [ 632.907164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 649.813922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 649.828638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 650.962066] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 650.977298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 651.024714] Lustre: lustre-MDT0000: new disk, initializing [ 651.071402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 651.081718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 652.920916] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 658.343779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 658.412883] Lustre: 45886:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 658.447568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 658.451648] Lustre: Skipped 1 previous similar message [ 658.502818] Lustre: lustre-MDT0001: new disk, initializing [ 658.558658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 658.581574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 658.589739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 660.739552] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 666.194749] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 668.728272] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 671.870338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 671.993971] Lustre: lustre-OST0000: new disk, initializing [ 671.996567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 671.999638] Lustre: 47853:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 672.040930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 674.727356] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 678.414688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 678.424743] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 678.487023] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 679.661916] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 681.618579] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 681.706431] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 683.584623] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 683.682146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 688.610115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 688.611084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 688.615625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 688.627098] Lustre: Skipped 1 previous similar message [ 690.732177] Lustre: server umount lustre-OST0000 complete [ 693.730050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 693.730956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 693.737518] Lustre: Skipped 1 previous similar message [ 693.742154] Lustre: Skipped 1 previous similar message [ 698.732211] Lustre: server umount lustre-MDT0000 complete [ 700.291372] LustreError: 45877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361638 with bad export cookie 11055155544123517033 [ 700.294378] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 700.295763] LustreError: 45877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 700.423617] Lustre: server umount lustre-MDT0001 complete [ 704.235175] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 00:27:20 (1782361640) [ 707.045934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 707.301948] LustreError: 50309:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 707.312531] LustreError: 50309:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 707.344527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 709.093180] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 712.668038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 712.672414] LustreError: 50310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 712.848121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 714.523337] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 717.769977] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 720.282652] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 723.319048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 723.465394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 726.110370] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 729.155357] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 741.289253] LustreError: 52856:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 741.707648] Lustre: Modifying parameter general.timeout=1 in log params [ 742.951543] Lustre: Modifying parameter general.timeout=4 in log params [ 742.954206] Lustre: Skipped 2 previous similar messages [ 745.134909] Lustre: Modifying parameter general.timeout=10 in log params [ 745.137502] Lustre: Skipped 5 previous similar messages [ 749.373603] Lustre: Modifying parameter general.timeout=21 in log params [ 749.376188] Lustre: Skipped 10 previous similar messages [ 757.589058] Lustre: Modifying parameter general.timeout=41 in log params [ 757.591797] Lustre: Skipped 19 previous similar messages [ 773.748073] Lustre: Modifying parameter general.timeout=82 in log params [ 773.751063] Lustre: Skipped 40 previous similar messages [ 796.958381] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 00:28:53 (1782361733) [ 798.226681] Lustre: 59761:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 799.827891] Lustre: 59957:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 799.834404] Lustre: 59957:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 1 previous similar message [ 802.227709] Lustre: 60252:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 802.230857] Lustre: 60252:0:(mgs_llog.c:1447:mgs_modify_param()) Skipped 2 previous similar messages [ 806.648540] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=change rule4 rate=50 in log params [ 806.651846] Lustre: Skipped 62 previous similar messages [ 814.324556] Lustre: 61601:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 814.332664] LustreError: 61601:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 816.173990] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 00:29:12 (1782361752) [ 821.219473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 821.224632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 821.232216] Lustre: Skipped 2 previous similar messages [ 821.237196] Lustre: Skipped 2 previous similar messages [ 826.336173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 826.338711] Lustre: Skipped 2 previous similar messages [ 826.992197] Lustre: server umount lustre-MDT0000 complete [ 828.665381] LustreError: 50290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361766 with bad export cookie 11055155544123517901 [ 828.669109] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 828.670783] LustreError: 50290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 828.813796] Lustre: server umount lustre-MDT0001 complete [ 840.829131] Lustre: server umount lustre-OST0000 complete [ 852.737437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.870700] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 852.972342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 854.683395] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 858.777344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 858.828688] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 858.843967] Lustre: 64125:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 860.892566] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 864.516584] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 867.239746] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 870.507713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 870.593551] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 873.291585] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 876.364417] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 882.415898] Lustre: Setting parameter lustre.mdt.lustre-MDT0001.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 882.419693] Lustre: Skipped 8 previous similar messages [ 882.437751] Lustre: 66690:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 888.803458] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 888.803578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 888.809144] LustreError: Skipped 2 previous similar messages [ 888.815978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 888.819601] Lustre: Skipped 1 previous similar message [ 899.039143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 899.041254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 899.052482] Lustre: Skipped 3 previous similar messages [ 899.119951] Lustre: server umount lustre-OST0000 complete [ 904.161015] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 904.165659] Lustre: Skipped 2 previous similar messages [ 906.921238] Lustre: server umount lustre-MDT0000 complete [ 908.585797] LustreError: 65985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361846 with bad export cookie 11055155544123520358 [ 908.589504] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 908.591375] LustreError: 65985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 908.729420] Lustre: server umount lustre-MDT0001 complete [ 913.216880] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 914.443554] Key type lgssc unregistered [ 914.587503] LNet: 67899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.590468] LNetError: 67899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.599375] LNet: Removed LNI 192.168.202.104@tcp [ 914.980707] Key type .llcrypt unregistered [ 914.982576] Key type ._llcrypt unregistered [ 925.894384] Key type ._llcrypt registered [ 925.895576] Key type .llcrypt registered [ 925.937755] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 926.426504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 926.438587] alg: No test for adler32 (adler32-zlib) [ 927.315193] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 927.434176] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 929.031470] Key type lgssc registered [ 929.453954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 932.800699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 932.810717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.018758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 935.599629] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 939.176052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 939.448264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 941.044657] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 943.937808] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 946.194610] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 949.164349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 949.425835] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 951.948331] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 955.064200] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 964.859760] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 969.185081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.186859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 969.191023] Lustre: Skipped 2 previous similar messages [ 969.195408] Lustre: Skipped 2 previous similar messages [ 973.919850] Lustre: server umount lustre-MDT0000 complete [ 974.305262] LustreError: 69320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 974.314490] LustreError: 69320:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 975.455063] LustreError: 69283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361913 with bad export cookie 10070773900391843611 [ 975.456782] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 975.460821] LustreError: 69283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 975.592996] Lustre: server umount lustre-MDT0001 complete [ 986.334287] Lustre: server umount lustre-OST0000 complete [ 998.288499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 998.460736] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 998.474989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 998.594391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1000.385310] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1004.279390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1004.337302] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1004.352545] Lustre: 74169:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1004.559424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1006.276406] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1009.273438] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1011.326932] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1014.271474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1014.370200] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1014.545995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1017.098363] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1020.319034] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1021.103364] Lustre: 76740:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1021.958062] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1024.998558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1024.998980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1025.005749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1025.011066] Lustre: Skipped 1 previous similar message [ 1029.716802] Lustre: server umount lustre-OST0000 complete [ 1035.233408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.233426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1035.234456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1035.239225] Lustre: Skipped 2 previous similar messages [ 1035.242075] LustreError: Skipped 1 previous similar message [ 1037.272836] Lustre: server umount lustre-MDT0000 complete [ 1038.579057] LustreError: 74159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361976 with bad export cookie 10070773900391844423 [ 1038.581528] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1038.583296] LustreError: 74159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1038.714162] Lustre: server umount lustre-MDT0001 complete [ 1042.676911] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 1043.772450] Key type lgssc unregistered [ 1043.906516] LNet: 77950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.910329] LNetError: 77950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.919365] LNet: Removed LNI 192.168.202.104@tcp [ 1044.281756] Key type .llcrypt unregistered [ 1044.283243] Key type ._llcrypt unregistered [ 1054.725353] Key type ._llcrypt registered [ 1054.726697] Key type .llcrypt registered [ 1054.768616] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1055.079802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1055.114039] alg: No test for adler32 (adler32-zlib) [ 1055.988985] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 1056.098915] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 1057.687105] Key type lgssc registered [ 1058.194179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1061.474673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1061.485472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1062.662166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1064.243142] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1067.592491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1067.824979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1069.357394] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1071.925678] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1073.899099] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1076.413379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1076.648372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1078.931596] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1081.800606] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1094.611058] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1095.297374] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1095.300823] Lustre: Skipped 1 previous similar message [ 1096.325992] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1096.328400] Lustre: Skipped 2 previous similar messages [ 1098.496882] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1098.499149] Lustre: Skipped 5 previous similar messages [ 1102.527451] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1102.530224] Lustre: Skipped 10 previous similar messages [ 1110.781369] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1110.783545] Lustre: Skipped 23 previous similar messages [ 1126.873598] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1126.875769] Lustre: Skipped 44 previous similar messages [ 1133.167297] Lustre: 87428:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1134.163861] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 00:34:30 (1782362070) [ 1134.777347] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1135.442097] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 00:34:32 (1782362072) [ 1139.169273] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1139.170177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.173064] LustreError: Skipped 1 previous similar message [ 1139.180161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1139.183458] Lustre: Skipped 1 previous similar message [ 1142.610218] Lustre: server umount lustre-OST0000 complete [ 1144.288776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.291051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1144.293054] Lustre: Skipped 2 previous similar messages [ 1149.408149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1149.410755] Lustre: Skipped 2 previous similar messages [ 1150.103784] Lustre: server umount lustre-MDT0000 complete [ 1151.534386] LustreError: 79320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782362089 with bad export cookie 11369980097121563640 [ 1151.537352] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1151.538586] LustreError: 79320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1151.678593] Lustre: server umount lustre-MDT0001 complete [ 1155.397218] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 1156.496498] Key type lgssc unregistered [ 1156.621412] LNet: 88674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.624263] LNetError: 88674:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.635319] LNet: Removed LNI 192.168.202.104@tcp [ 1156.982377] Key type .llcrypt unregistered [ 1156.984191] Key type ._llcrypt unregistered [ 1158.498832] Key type ._llcrypt registered [ 1158.500672] Key type .llcrypt registered [ 1158.546357] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1160.027923] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules [ 1160.520809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1160.535150] alg: No test for adler32 (adler32-zlib) [ 1161.403361] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 1161.497450] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 1163.079155] Key type lgssc registered [ 1163.522950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1163.953137] LustreError: 89827:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 1171.780606] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1175.249520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1185.068454] LustreError: 89827:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 1185.077392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1186.260825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1187.622028] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1190.418694] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 00:35:27 (1782362127) [ 1192.015403] LustreError: 90883:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1192.019373] LustreError: 90883:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1192.022717] LustreError: 90883:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 0, failed: rc = -5 [ 1192.154638] Lustre: server umount lustre-MDT0000 complete [ 1197.178782] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 1198.230476] Key type lgssc unregistered [ 1198.362724] LNet: 92375:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1198.365216] LNetError: 92375:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1198.377300] LNet: Removed LNI 192.168.202.104@tcp [ 1198.699753] Key type .llcrypt unregistered [ 1198.701180] Key type ._llcrypt unregistered [ 1205.831633] Key type ._llcrypt registered [ 1205.833018] Key type .llcrypt registered [ 1205.879960] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1206.286996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1206.358792] alg: No test for adler32 (adler32-zlib) [ 1207.224712] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 1207.329837] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 1208.911191] Key type lgssc registered [ 1209.280302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1211.947391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1211.954386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.150808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1214.618232] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1217.736806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.992187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1219.642598] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1222.427621] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1224.282145] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1226.473140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1226.675315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1228.563866] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1230.938404] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1261.645916] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 00:36:38 (1782362198) [ 1262.188378] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 1262.727640] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 00:36:39 (1782362199) [ 1264.095806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1264.098401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.102711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1264.608709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.610099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1264.613166] Lustre: Skipped 1 previous similar message [ 1269.728520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1269.730212] Lustre: Skipped 3 previous similar messages [ 1269.774358] Lustre: server umount lustre-MDT0000 complete [ 1270.895080] LustreError: 93550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782362208 with bad export cookie 3745271549659642438 [ 1270.897725] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1270.899287] LustreError: 93550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1271.004863] Lustre: server umount lustre-MDT0001 complete [ 1282.555784] Lustre: server umount lustre-OST0000 complete [ 1286.141240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1286.280440] LustreError: 97859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.287028] LustreError: 97859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1286.331842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.646428] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1290.263538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1290.487781] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1291.788991] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1294.094838] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1295.750985] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1299.715796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1299.786609] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1299.789127] LustreError: 97847:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1299.797952] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1299.800585] LustreError: 99941:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1299.802613] LustreError: 99941:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1299.804385] LustreError: 99941:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1299.825579] Lustre: server umount lustre-OST0000 complete [ 1299.826939] LustreError: 99941:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1302.526767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1302.584790] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 1302.587273] LustreError: 97847:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 1302.590871] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 1302.595540] LustreError: 100401:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 1302.599269] LustreError: 100401:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 1302.601715] LustreError: 100401:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 1302.622405] Lustre: server umount lustre-OST0000 complete [ 1302.623816] LustreError: 100401:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 1305.601845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1305.663826] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1305.665862] Lustre: Found index 0 for lustre-OST0000, updating log [ 1305.668231] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 1305.694141] Lustre: lustre-OST0000: new disk, initializing [ 1305.695850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1305.698142] Lustre: 100933:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1305.780922] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1307.747919] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1309.031085] Lustre: Failing over lustre-OST0000 [ 1309.078188] Lustre: server umount lustre-OST0000 complete [ 1311.200815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1311.200871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1311.201458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1311.205993] Lustre: Skipped 1 previous similar message [ 1315.296136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1315.297922] Lustre: Skipped 2 previous similar messages [ 1316.497346] Lustre: server umount lustre-MDT0000 complete [ 1317.684203] LustreError: 97839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782362255 with bad export cookie 3745271549659645175 [ 1317.684875] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1317.687509] LustreError: 97839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1317.794902] Lustre: server umount lustre-MDT0001 complete [ 1320.680290] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 00:37:37 (1782362257) [ 1325.467277] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1329.172905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1329.307650] LustreError: 103272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1329.334952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1330.620478] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1333.372277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1334.933667] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1335.904700] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 1338.067931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1340.220772] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1343.129654] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1343.163075] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 1343.197852] Lustre: lustre-OST0001: new disk, initializing [ 1343.199985] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1343.203626] Lustre: 105589:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1344.711018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1344.715800] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1344.748881] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1345.377338] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1348.465535] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1355.084786] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1356.540477] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.4@tcp (stopping) [ 1358.815809] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 1358.818950] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1362.448030] Lustre: server umount lustre-MDT0001 complete [ 1365.392872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1365.425153] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1365.426890] Lustre: Found index 1 for lustre-MDT0001, updating log [ 1365.439470] Lustre: 103261:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1365.472992] LustreError: 103272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1365.479168] LustreError: 103272:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1365.661050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1365.663578] Lustre: Skipped 3 previous similar messages [ 1366.939640] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1367.015515] LustreError: 107364:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 1367.019120] LustreError: 107364:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 1367.023558] LustreError: 103265:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 1367.051410] LustreError: 107374:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.104@tcp: cfg command failed: rc = -17 [ 1367.054115] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 1367.054115] [ 1369.952659] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 00:38:26 (1782362306) [ 1371.109088] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1375.260380] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1380.967629] Lustre: 109196:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1388.669881] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1389.780608] Lustre: 110328:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1437.151646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1437.153586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1437.156344] Lustre: Skipped 2 previous similar messages [ 1437.157599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1437.159752] Lustre: Skipped 5 previous similar messages [ 1440.023820] Lustre: server umount lustre-MDT0000 complete [ 1441.212637] LustreError: 103255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782362379 with bad export cookie 3745271549659646043 [ 1441.214027] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1441.216216] LustreError: 103255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1441.321153] Lustre: server umount lustre-MDT0001 complete [ 1452.997168] Lustre: server umount lustre-OST0000 complete [ 1464.406941] Lustre: server umount lustre-OST0001 complete [ 1466.129120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1469.024632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1471.627563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1474.600982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 1480.264494] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1483.420218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1483.426133] Lustre: lustre-MDT0000: reset Object Index mappings [ 1483.552995] LustreError: 114247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1483.572533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1484.790256] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1487.455382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1487.463572] Lustre: lustre-MDT0001: reset Object Index mappings [ 1488.975165] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1489.862959] Lustre: 115393:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 1491.919651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1493.155415] LustreError: 115748:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1493.160056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 1493.968542] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1496.678141] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1497.897410] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:157 to 0x2c0000401:193) [ 1497.898887] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:199 to 0x280000400:225) [ 1497.927966] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 1498.640086] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1501.709273] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1502.837093] Lustre: 117265:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 1516.545678] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 00:40:53 (1782362453) [ 1549.281418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1549.284027] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1549.284685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1549.284691] Lustre: Skipped 3 previous similar messages [ 1549.293739] Lustre: Skipped 5 previous similar messages [ 1551.912440] Lustre: server umount lustre-MDT0000 complete [ 1553.069126] LustreError: 114995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782362490 with bad export cookie 3745271549659692810 [ 1553.072289] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1553.073364] LustreError: 114995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1554.399921] LustreError: 115772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1554.404303] LustreError: 115772:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1559.345436] Lustre: server umount lustre-MDT0001 complete [ 1566.652865] Lustre: server umount lustre-OST0000 complete [ 1582.047115] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1582.109732] Lustre: server umount lustre-OST0001 complete [ 1584.083564] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 1587.193618] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1598.815335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1598.895041] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1598.904483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1598.935894] Lustre: lustre-MDT0000: new disk, initializing [ 1598.960743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1600.358441] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1603.274946] Lustre: server umount lustre-MDT0000 complete [ 1606.251184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1606.349719] Lustre: Found index 0 for lustre-MDT0000, updating log [ 1606.355727] Lustre: 121527:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1606.360900] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 1607.865740] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1609.612598] Lustre: server umount lustre-MDT0000 complete [ 1612.283543] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 00:42:29 (1782362549) [ 1612.750083] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 1613.267475] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 00:42:30 (1782362550) [ 1618.341151] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1621.488254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1621.599571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1621.601524] Lustre: Skipped 5 previous similar messages [ 1622.823499] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1625.406226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1625.443688] Lustre: 123579:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1625.465158] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1625.467051] Lustre: Skipped 1 previous similar message [ 1625.497335] Lustre: lustre-MDT0001: new disk, initializing [ 1625.539202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1625.542019] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1626.744032] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1628.890910] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 1630.867680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1630.959438] Lustre: lustre-OST0000: new disk, initializing [ 1630.960954] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1630.963269] Lustre: 125152:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1631.733633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1631.736707] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1631.769586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1632.801169] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1636.731015] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1636.789350] Lustre: 126025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1638.531300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1638.535196] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1638.579980] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1638.747083] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1643.050865] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1644.127282] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 1779.447532] Lustre: 132398:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 1779.450608] Lustre: 132398:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 1779.453084] Lustre: 132398:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 1779.455578] Lustre: 132398:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 1779.457877] Lustre: 132398:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 1779.460660] Lustre: 132398:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1845.576101] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 00:46:22 (1782362782) [ 1859.042441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.043929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1859.047093] Lustre: Skipped 6 previous similar messages [ 1859.050689] Lustre: Skipped 10 previous similar messages [ 1863.413934] Lustre: server umount lustre-MDT0000 complete [ 1864.161088] LustreError: 126825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1864.166085] LustreError: 126825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1864.699295] LustreError: 123569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782362802 with bad export cookie 3745271549659745170 [ 1864.703922] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1864.704032] LustreError: 123569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1879.007050] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1879.120658] Lustre: server umount lustre-MDT0001 complete [ 1886.533263] Lustre: server umount lustre-OST0000 complete [ 1902.047128] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1902.114135] Lustre: server umount lustre-OST0001 complete [ 1904.342323] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 1907.712867] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 1919.808835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1919.910027] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1919.921653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1919.923236] Lustre: Skipped 1 previous similar message [ 1919.961942] Lustre: lustre-MDT0000: new disk, initializing [ 1919.963314] Lustre: Skipped 1 previous similar message [ 1919.984775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1919.986596] Lustre: Skipped 3 previous similar messages [ 1919.991165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1921.325215] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1925.619424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.675937] Lustre: 135411:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1925.807393] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1927.336347] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1931.276991] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1933.215729] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1935.644655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1935.770389] Lustre: 137379:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1937.028314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1937.032804] Lustre: Skipped 1 previous similar message [ 1937.035682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1937.067359] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1937.906228] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 1941.725345] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1945.020910] Lustre: lustre-MDD0000: changelog on [ 1945.953780] Lustre: lustre-MDD0001: changelog on [ 2286.605196] Lustre: lustre-MDD0001: changelog off [ 2287.243658] Lustre: 135419:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 2287.552430] Lustre: lustre-MDD0000: changelog off [ 2290.657258] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2290.658948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2290.659741] LustreError: Skipped 1 previous similar message [ 2290.660462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2290.663214] Lustre: Skipped 3 previous similar messages [ 2290.664471] Lustre: Skipped 4 previous similar messages [ 2295.187969] Lustre: server umount lustre-OST0000 complete [ 2299.872031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2302.527429] Lustre: server umount lustre-MDT0000 complete [ 2303.776810] LustreError: 137378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363241 with bad export cookie 3745271549660507064 [ 2303.780854] LustreError: 137378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2303.781994] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2303.899825] Lustre: server umount lustre-MDT0001 complete [ 2307.824449] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 2308.816531] Key type lgssc unregistered [ 2308.940344] LNet: 145835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2308.943180] LNetError: 145835:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2308.955359] LNet: Removed LNI 192.168.202.104@tcp [ 2309.283137] Key type .llcrypt unregistered [ 2309.285105] Key type ._llcrypt unregistered [ 2315.811597] Key type ._llcrypt registered [ 2315.813057] Key type .llcrypt registered [ 2315.854947] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 2322.124917] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 2322.513715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2322.524113] alg: No test for adler32 (adler32-zlib) [ 2323.387438] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 2323.475961] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 2325.055154] Key type lgssc registered [ 2325.429676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2336.557558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2336.564347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2337.648945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2337.659116] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2337.697283] Lustre: lustre-MDT0000: new disk, initializing [ 2337.717171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2337.722795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2339.050180] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2343.142634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2343.180937] Lustre: 149007:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2343.202409] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2343.204265] Lustre: Skipped 1 previous similar message [ 2343.239070] Lustre: lustre-MDT0001: new disk, initializing [ 2343.270518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2343.281541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2343.285344] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2344.754629] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2348.474207] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2350.261843] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2352.831856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2352.946054] Lustre: lustre-OST0000: new disk, initializing [ 2352.947607] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2352.949801] Lustre: 150974:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2352.979301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2353.789442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2353.793740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2353.842394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2354.902220] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2358.626343] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2360.841591] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_module obdecho/obdecho [ 2361.454168] LustreError: 151977:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 2361.459676] LustreError: 151977:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 2362.091790] LustreError: 152073:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 2368.991885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2368.994338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2368.998766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2369.504358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2369.506590] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2369.509774] Lustre: Skipped 1 previous similar message [ 2371.480657] Lustre: server umount lustre-MDT0000 complete [ 2372.709642] LustreError: 149002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363310 with bad export cookie 13592286214631700921 [ 2372.713424] LustreError: 149002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2372.713916] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2372.829570] Lustre: server umount lustre-MDT0001 complete [ 2384.343951] Lustre: server umount lustre-OST0000 complete [ 2390.991783] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 2394.020030] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 2405.460490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.548082] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2405.557739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2405.598550] Lustre: lustre-MDT0000: new disk, initializing [ 2405.619183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2405.624932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2407.007697] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2411.329647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2411.370026] Lustre: 155777:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2411.397756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2411.399906] Lustre: Skipped 1 previous similar message [ 2411.442262] Lustre: lustre-MDT0001: new disk, initializing [ 2411.476027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2411.487991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2411.491142] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2412.780738] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2416.533610] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2418.354394] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2420.603235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2420.711315] Lustre: lustre-OST0000: new disk, initializing [ 2420.712852] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2420.715328] Lustre: 157743:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2420.749926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2422.655923] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2422.772376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2422.776959] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2422.808641] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2426.297815] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2427.652017] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2427.720239] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2429.051577] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2429.122396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2431.822921] Lustre: server umount lustre-OST0000 complete [ 2437.087994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2437.090530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2437.094741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2437.096581] Lustre: Skipped 2 previous similar messages [ 2443.232196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2443.234127] Lustre: Skipped 3 previous similar messages [ 2447.327116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2447.391981] Lustre: server umount lustre-MDT0000 complete [ 2448.159099] Lustre: 147238:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782363370/real 1782363370] req@000000004140a64c x1868943338825216/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782363386 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2448.167397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.173165] Lustre: Skipped 1 previous similar message [ 2448.638349] LustreError: 155771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363386 with bad export cookie 13592286214631701866 [ 2448.642129] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.644640] LustreError: 155771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2448.762768] Lustre: server umount lustre-MDT0001 complete [ 2449.967081] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 00:56:26 (1782363386) [ 2450.547297] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 2451.172699] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 00:56:27 (1782363387) [ 2453.786159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2453.935961] LustreError: 160154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.941583] LustreError: 160154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2453.959147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2455.382716] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2458.186886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2459.626761] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2462.068458] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2463.775105] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2466.028963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2468.061594] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2470.424702] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2483.683712] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2483.687038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2483.688074] Lustre: Skipped 2 previous similar messages [ 2483.690931] Lustre: Skipped 1 previous similar message [ 2488.798973] Lustre: server umount lustre-MDT0000 complete [ 2488.800287] LustreError: 160173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2490.045291] LustreError: 160136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363427 with bad export cookie 13592286214631702734 [ 2490.049108] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2490.049747] LustreError: 160136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2491.125607] LustreError: 160149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2491.130544] LustreError: 160149:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2493.920147] LustreError: 160150:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2493.920433] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2493.931970] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2493.933710] Lustre: Skipped 3 previous similar messages [ 2496.273224] Lustre: server umount lustre-MDT0001 complete [ 2498.538296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2503.577810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2503.655388] LustreError: 164015:0:(llog.c:1646:llog_backup()) MGC192.168.202.104@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 2503.657894] Lustre: 164015:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.104@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 2511.605087] LustreError: 164020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.610318] LustreError: 164020:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2522.869149] LustreError: 164021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.874255] LustreError: 164021:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2539.999914] LustreError: 164022:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.004684] LustreError: 164022:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 2565.087432] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 2565.089594] Lustre: 164015:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.202.104@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 2573.044987] LustreError: 164021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.051795] LustreError: 164021:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 19 previous similar messages [ 2605.545178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2605.548350] Lustre: Skipped 2 previous similar messages [ 2607.175481] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2609.637347] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2610.315720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2634.207629] LustreError: 147235:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000188d1400 x1868943338869760/t0(0) o250->MGC192.168.202.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2635.708812] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2638.222145] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2638.764908] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2639.333155] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2640.805194] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2643.609853] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 2654.693026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2654.693355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2654.696697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2654.696702] Lustre: Skipped 1 previous similar message [ 2654.701801] Lustre: Skipped 2 previous similar messages [ 2655.384780] Lustre: server umount lustre-MDT0000 complete [ 2656.991255] LustreError: 160889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363594 with bad export cookie 13592286214631709181 [ 2656.994764] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2656.995991] LustreError: 160889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2657.113313] Lustre: server umount lustre-MDT0001 complete [ 2668.910521] Lustre: server umount lustre-OST0000 complete [ 2679.366935] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 2684.399353] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 2702.938741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2703.042429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2703.057980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2703.097850] Lustre: lustre-MDT0000: new disk, initializing [ 2703.129379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2703.132735] Lustre: Skipped 1 previous similar message [ 2703.145494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2705.156825] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2711.164065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2711.232133] Lustre: 169094:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2711.411279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2713.672518] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2718.088537] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2720.157117] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2722.880446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2722.998190] Lustre: 171060:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2725.264632] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2729.479663] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2729.537849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2729.576347] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2731.496992] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2731.588228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2733.255835] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2733.331397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2734.559847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2734.561064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2734.564131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2734.567573] Lustre: Skipped 1 previous similar message [ 2734.570262] Lustre: Skipped 2 previous similar messages [ 2736.156095] Lustre: server umount lustre-OST0000 complete [ 2743.771093] Lustre: server umount lustre-MDT0000 complete [ 2744.800047] LustreError: 169105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2744.806175] LustreError: 169105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 46 previous similar messages [ 2745.165982] LustreError: 169086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363683 with bad export cookie 13592286214631726009 [ 2745.168607] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2745.170286] LustreError: 169086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2745.298222] Lustre: server umount lustre-MDT0001 complete [ 2749.034484] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 01:01:25 (1782363685) [ 2751.894431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2753.709139] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2756.999251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2758.624847] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2761.708475] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2764.193631] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2767.137556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2769.863831] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2772.637418] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2787.148558] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 01:02:03 (1782363723) [ 2788.834183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2794.255640] Lustre: server umount lustre-OST0000 complete [ 2798.047752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2798.051553] LustreError: Skipped 1 previous similar message [ 2799.072102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2799.073141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2799.074154] Lustre: Skipped 7 previous similar messages [ 2799.078815] Lustre: Skipped 5 previous similar messages [ 2801.748502] Lustre: server umount lustre-MDT0000 complete [ 2803.104373] LustreError: 173507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363741 with bad export cookie 13592286214631726877 [ 2803.107308] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2803.109028] LustreError: 173507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2803.237630] Lustre: server umount lustre-MDT0001 complete [ 2807.006080] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 2808.053642] Key type lgssc unregistered [ 2808.175247] LNet: 177223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2808.177690] LNetError: 177223:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2808.187290] LNet: Removed LNI 192.168.202.104@tcp [ 2808.495939] Key type .llcrypt unregistered [ 2808.497533] Key type ._llcrypt unregistered [ 2815.796116] Key type ._llcrypt registered [ 2815.797680] Key type .llcrypt registered [ 2815.831955] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 2816.132806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2816.162523] alg: No test for adler32 (adler32-zlib) [ 2817.016541] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 2817.102377] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 2818.687136] Key type lgssc registered [ 2819.062399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2821.634164] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2821.639231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2837.087352] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2842.207326] LustreError: 178443:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.104@tcp: failed processing log, type 4: rc = -110 [ 2847.327545] LustreError: 178443:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2847.331354] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 2847.333179] LustreError: MGC192.168.202.104@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 2847.336460] LustreError: 178443:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 2847.338961] LustreError: 178443:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 2847.341318] LustreError: 178443:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2847.360760] Lustre: server umount lustre-OST0000 complete [ 2847.362184] LustreError: 178443:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 2850.170227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2851.317640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2852.773524] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2855.642624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2855.767615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2857.110820] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2859.464385] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2861.264881] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2863.436306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2863.500196] LustreError: 180754:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 2863.503423] Lustre: 180754:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.104@tcp: invalid local config log lustre-OST0000: rc = -22 [ 2863.551471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2865.417678] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2867.861892] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2870.753289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2870.755598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2870.758743] Lustre: Skipped 2 previous similar messages [ 2870.760782] Lustre: Skipped 1 previous similar message [ 2874.708761] Lustre: server umount lustre-MDT0000 complete [ 2875.873935] LustreError: 178940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2875.878330] LustreError: 178940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2875.911398] LustreError: 178923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363813 with bad export cookie 11058485386834932770 [ 2875.913850] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2875.915644] LustreError: 178923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2876.037792] Lustre: server umount lustre-MDT0001 complete [ 2887.580560] Lustre: server umount lustre-OST0000 complete [ 2890.653909] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 2894.180058] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 2906.191696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.273560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2906.283080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2906.318210] Lustre: lustre-MDT0000: new disk, initializing [ 2906.356599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2906.362142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2907.679926] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2911.956742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.999731] Lustre: 184269:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2912.021288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2912.022814] Lustre: Skipped 1 previous similar message [ 2912.061827] Lustre: lustre-MDT0001: new disk, initializing [ 2912.097866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2912.113700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2912.118159] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2913.558793] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2917.259673] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2919.084666] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2921.281804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2921.374121] Lustre: lustre-OST0000: new disk, initializing [ 2921.376855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2921.379705] Lustre: 186239:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2921.412133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2923.342790] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2926.867691] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2928.235782] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2929.144630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2929.147680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2929.174845] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2929.319697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2930.669794] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2930.748976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2934.242115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2934.242973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2934.244763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2934.247575] Lustre: Skipped 1 previous similar message [ 2934.250572] Lustre: Skipped 1 previous similar message [ 2937.484222] Lustre: server umount lustre-OST0000 complete [ 2939.360752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2939.361543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2939.366302] Lustre: Skipped 1 previous similar message [ 2939.369102] Lustre: Skipped 1 previous similar message [ 2944.480331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2944.483126] Lustre: Skipped 1 previous similar message [ 2944.795821] Lustre: server umount lustre-MDT0000 complete [ 2945.979310] LustreError: 185068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363883 with bad export cookie 11058485386834933456 [ 2945.982571] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2945.983245] LustreError: 185068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2946.088044] Lustre: server umount lustre-MDT0001 complete [ 2949.023213] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 01:04:45 (1782363885) [ 2954.753642] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 2956.047931] Key type lgssc unregistered [ 2956.163423] LNet: 189191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2956.165949] LNetError: 189191:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2956.174303] LNet: Removed LNI 192.168.202.104@tcp [ 2956.453317] Key type .llcrypt unregistered [ 2956.454337] Key type ._llcrypt unregistered [ 2963.667616] Key type ._llcrypt registered [ 2963.669414] Key type .llcrypt registered [ 2963.704810] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 2964.102087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2964.113841] alg: No test for adler32 (adler32-zlib) [ 2964.967307] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 2965.051422] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 2966.631131] Key type lgssc registered [ 2966.987098] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2969.336857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2969.342460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2970.486277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2971.725952] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2974.341978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.452485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2975.783556] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2978.035545] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2979.699819] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2982.173672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2982.250128] Lustre: *** cfs_fail_loc=131e, val=0*** [ 2982.251595] LustreError: 192247:0:(llog.c:1687:llog_backup()) MGC192.168.202.104@tcp: failed to backup log lustre-OST0000: rc = -28 [ 2982.253908] Lustre: 192247:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.104@tcp: can't backup local config lustre-OST0000: rc = -28 [ 2982.292066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2984.141321] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 2986.369718] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2989.537593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2989.539288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2989.543075] Lustre: Skipped 1 previous similar message [ 2994.659468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2994.661855] Lustre: Skipped 4 previous similar messages [ 2999.777840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3001.311127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3001.379743] Lustre: server umount lustre-MDT0000 complete [ 3002.574275] LustreError: 190365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363940 with bad export cookie 15507053445951516184 [ 3002.576695] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3002.577441] LustreError: 190365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3002.709859] Lustre: server umount lustre-MDT0001 complete [ 3014.175424] Lustre: server umount lustre-OST0000 complete [ 3017.195251] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 3020.305873] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3031.563176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.641614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3031.652281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3031.694208] Lustre: lustre-MDT0000: new disk, initializing [ 3031.713487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3031.718836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3032.979941] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3036.901311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.937648] Lustre: 195765:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3036.953883] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3036.954936] Lustre: Skipped 1 previous similar message [ 3036.956072] LustreError: 196544:0:(llog.c:1687:llog_backup()) MGC192.168.202.104@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 3036.958312] LustreError: 196544:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 3036.959969] Lustre: 196544:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.104@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 3036.966382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3036.967875] Lustre: Skipped 1 previous similar message [ 3037.000926] Lustre: lustre-MDT0001: new disk, initializing [ 3037.039201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3037.051518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3037.054447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3038.292899] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3041.796901] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3043.569172] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3045.880026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3045.978961] Lustre: *** cfs_fail_loc=131e, val=0*** [ 3045.980106] Lustre: Skipped 1 previous similar message [ 3045.981240] LustreError: 197706:0:(llog.c:1687:llog_backup()) MGC192.168.202.104@tcp: failed to backup log lustre-OST0000: rc = -28 [ 3045.984128] LustreError: 197706:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 3045.986030] Lustre: 197706:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.104@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 3045.988684] Lustre: 197706:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 3045.996242] Lustre: lustre-OST0000: new disk, initializing [ 3045.997855] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3046.000339] Lustre: 197730:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3046.027458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.858950] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3048.185452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3048.188338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3048.219772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3051.561935] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3052.942484] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3053.014020] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3054.334378] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3054.418205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3057.241249] Lustre: server umount lustre-OST0000 complete [ 3062.751951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3062.753969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3062.757182] Lustre: Skipped 1 previous similar message [ 3062.759094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3062.761260] Lustre: Skipped 2 previous similar messages [ 3063.777257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3068.896478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3068.899142] Lustre: Skipped 3 previous similar messages [ 3072.991122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3073.051802] Lustre: server umount lustre-MDT0000 complete [ 3074.016844] LustreError: 195775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3074.022102] LustreError: 195775:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3074.497959] LustreError: 195757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364012 with bad export cookie 15507053445951516870 [ 3074.504196] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3084.832450] Lustre: server umount lustre-MDT0001 complete [ 3087.775069] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 01:07:04 (1782364024) [ 3093.056947] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3096.410096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3096.547976] LustreError: 200854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3096.560067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3097.874744] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3100.593018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3102.087474] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3103.039355] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3105.113212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3107.073029] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3109.949065] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3110.028985] Lustre: lustre-OST0001: new disk, initializing [ 3110.030884] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3110.033354] Lustre: 203160:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3111.356605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3111.359542] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3111.392052] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3111.922874] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3117.656879] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3124.505830] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3125.114285] Lustre: DEBUG MARKER: ADD OST3 [ 3126.766181] loop: module loaded [ 3126.784532] loop0: detected capacity change from 0 to 204800000 [ 3126.859324] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3126.863287] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3126.867121] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 3128.855389] loop0: detected capacity change from 0 to 204800000 [ 3130.698946] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3130.773614] Lustre: lustre-OST0003: new disk, initializing [ 3130.775522] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 3130.778081] Lustre: 204893:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3130.817960] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 3130.820400] Lustre: Skipped 3 previous similar messages [ 3131.804851] Lustre: DEBUG MARKER: STOP OST3 [ 3132.178901] Lustre: *** cfs_fail_loc=2109, val=2*** [ 3132.213630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 3132.216711] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 3132.226901] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 3132.512296] Lustre: Failing over lustre-OST0003 [ 3132.551985] Lustre: server umount lustre-OST0003 complete [ 3132.612660] LustreError: 202339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.895891] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3134.239105] LustreError: 205195:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 3136.851297] Lustre: DEBUG MARKER: START OST3 again [ 3137.506371] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3137.509209] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3137.515388] LustreError: 202339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3137.520572] LustreError: 202339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3139.249522] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3139.333387] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 3140.834349] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3140.842843] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3140.842873] Lustre: lustre-OST0003: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3141.148114] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3154.427990] Lustre: Failing over lustre-OST0003 [ 3154.465958] Lustre: server umount lustre-OST0003 complete [ 3154.918230] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3154.922696] Lustre: Skipped 1 previous similar message [ 3154.924781] LustreError: 202525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3154.929397] LustreError: 202525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3165.156885] LustreError: 203949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3165.164337] LustreError: 203949:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3167.199842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3167.202964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3167.207704] Lustre: Skipped 1 previous similar message [ 3167.209991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3173.024605] Lustre: server umount lustre-MDT0000 complete [ 3174.193337] LustreError: 201587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364112 with bad export cookie 15507053445951517738 [ 3174.197693] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3174.199424] LustreError: 201587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3180.523621] Lustre: server umount lustre-MDT0001 complete [ 3181.799435] Lustre: server umount lustre-OST0000 complete [ 3183.034989] Lustre: server umount lustre-OST0001 complete [ 3185.049986] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 3187.526875] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3200.713866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3200.790255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3200.800887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3200.837108] Lustre: lustre-MDT0000: new disk, initializing [ 3200.857692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3200.860991] Lustre: Skipped 1 previous similar message [ 3200.867428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3202.197099] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3206.278642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3206.315728] Lustre: 210300:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3206.417516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3207.804115] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3211.513535] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3213.268197] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3215.429825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3215.536074] Lustre: 212268:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3217.272555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3217.276448] Lustre: Skipped 1 previous similar message [ 3217.278928] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3217.314265] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3217.478394] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3221.000171] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3222.398185] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3222.468168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3223.763897] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3223.843733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3227.617487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3227.619286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.619849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3227.619853] Lustre: Skipped 8 previous similar messages [ 3227.626704] Lustre: Skipped 6 previous similar messages [ 3230.607058] Lustre: server umount lustre-OST0000 complete [ 3232.224069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3232.226728] LustreError: Skipped 1 previous similar message [ 3237.976387] Lustre: server umount lustre-MDT0000 complete [ 3239.161106] LustreError: 212267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364177 with bad export cookie 15507053445951519985 [ 3239.164579] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3239.164631] LustreError: 212267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3239.276873] Lustre: server umount lustre-MDT0001 complete [ 3240.439784] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 01:09:37 (1782364177) [ 3245.276745] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 3248.255224] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3259.679750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3259.764102] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3259.774742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3259.776496] Lustre: Skipped 3 previous similar messages [ 3259.816474] Lustre: lustre-MDT0000: new disk, initializing [ 3259.818269] Lustre: Skipped 2 previous similar messages [ 3259.844974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3261.212118] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3265.259720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.300317] Lustre: 216868:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3265.417119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3266.733667] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3270.397778] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3272.112780] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3274.425915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3274.554001] Lustre: 218835:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3276.267743] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3276.541341] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3280.127131] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3281.616765] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3281.694292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3283.024978] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3283.095746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3286.498360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3286.498457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3286.501830] Lustre: Skipped 2 previous similar messages [ 3286.502536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3286.503912] LustreError: Skipped 1 previous similar message [ 3286.508398] Lustre: Skipped 6 previous similar messages [ 3289.805233] Lustre: server umount lustre-OST0000 complete [ 3297.108076] Lustre: server umount lustre-MDT0000 complete [ 3298.354843] LustreError: 216861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364236 with bad export cookie 15507053445951520867 [ 3298.359341] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3298.360925] LustreError: 216861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3298.509235] Lustre: server umount lustre-MDT0001 complete [ 3301.752231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3301.915990] LustreError: 221055:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3301.921567] LustreError: 221055:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3303.445141] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3307.422063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.918233] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3311.338633] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3313.104333] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3315.406051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3317.430064] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3320.119406] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3337.185758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3337.188036] LustreError: Skipped 1 previous similar message [ 3343.123428] Lustre: server umount lustre-OST0000 complete [ 3350.645846] Lustre: server umount lustre-MDT0000 complete [ 3352.052043] LustreError: 223651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364289 with bad export cookie 15507053445951521749 [ 3352.060342] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3352.062939] LustreError: 223651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3352.207684] Lustre: server umount lustre-MDT0001 complete [ 3454.203127] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 3455.249064] Key type lgssc unregistered [ 3455.376363] LNet: 224737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3455.378760] LNetError: 224737:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3455.388255] LNet: Removed LNI 192.168.202.104@tcp [ 3455.688158] Key type .llcrypt unregistered [ 3455.689502] Key type ._llcrypt unregistered [ 3464.241722] Key type ._llcrypt registered [ 3464.242952] Key type .llcrypt registered [ 3464.278128] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 3471.158460] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3471.535992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3471.563929] alg: No test for adler32 (adler32-zlib) [ 3472.416507] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 3472.509033] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 3474.095202] Key type lgssc registered [ 3474.598063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3489.478607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3489.486834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3490.576755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3490.586892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3490.620702] Lustre: lustre-MDT0000: new disk, initializing [ 3490.642894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3490.648631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3492.109925] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3497.334527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3497.393728] Lustre: 228060:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3497.430981] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3497.434920] Lustre: Skipped 1 previous similar message [ 3497.480990] Lustre: lustre-MDT0001: new disk, initializing [ 3497.516775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3497.534096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3497.539383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3499.295959] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3503.847088] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3506.289268] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3509.652230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3509.800367] Lustre: lustre-OST0000: new disk, initializing [ 3509.803312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3509.808815] Lustre: 230028:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3509.851223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3512.523389] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3515.410736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3515.416233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3515.454393] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3517.031605] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3519.000879] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3519.109968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3520.886896] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3520.968805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3525.605620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3525.606153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3525.608255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3525.614116] Lustre: Skipped 1 previous similar message [ 3525.619233] LustreError: Skipped 1 previous similar message [ 3530.726114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3530.730090] Lustre: Skipped 2 previous similar messages [ 3535.843118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3535.845818] Lustre: Skipped 1 previous similar message [ 3536.351115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3536.418500] Lustre: server umount lustre-OST0000 complete [ 3538.400793] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3538.405868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3538.414909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3540.961391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3546.080551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3546.083151] Lustre: Skipped 3 previous similar messages [ 3552.223287] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3552.324690] Lustre: server umount lustre-MDT0000 complete [ 3553.969887] LustreError: 228856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364491 with bad export cookie 4202237429534454035 [ 3553.976340] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3554.089351] Lustre: server umount lustre-MDT0001 complete [ 3560.341361] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3563.852833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3563.996976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3565.368699] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3568.296407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3568.430692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3569.731664] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3570.683838] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3572.809663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3574.769397] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3577.556024] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3577.619216] Lustre: lustre-OST0001: new disk, initializing [ 3577.621201] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3577.624493] Lustre: 235268:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3577.655277] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3577.657132] Lustre: Skipped 1 previous similar message [ 3579.549968] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3580.094559] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3580.098440] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3580.130911] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3585.005621] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3591.676226] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3595.562136] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 01:15:32 (1782364532) [ 3599.329301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3599.332560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3599.337321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3599.339130] Lustre: Skipped 2 previous similar messages [ 3602.331174] Lustre: server umount lustre-MDT0000 complete [ 3604.034096] LustreError: 234094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364541 with bad export cookie 4202237429534454917 [ 3604.039138] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3604.041463] LustreError: 234094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3604.178608] Lustre: server umount lustre-MDT0001 complete [ 3615.177276] Lustre: server umount lustre-OST0000 complete [ 3626.753185] Lustre: server umount lustre-OST0001 complete [ 3628.943168] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 3632.539271] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3646.197032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3646.282194] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3646.296218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3646.334209] Lustre: lustre-MDT0000: new disk, initializing [ 3646.353431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3646.359202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3647.639341] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3652.037792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3652.078242] Lustre: 239204:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3652.174094] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3653.478444] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3657.030600] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3658.809690] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3661.036290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3661.137353] Lustre: lustre-OST0000: new disk, initializing [ 3661.138923] Lustre: Skipped 1 previous similar message [ 3661.140656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3661.143075] Lustre: Skipped 2 previous similar messages [ 3661.145266] Lustre: 241171:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3662.903664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3662.906292] Lustre: Skipped 1 previous similar message [ 3662.907956] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3662.942427] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3663.061830] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3666.761331] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3668.172108] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3668.234905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3669.587045] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3669.650107] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3673.056511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3673.057760] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.058900] LustreError: Skipped 1 previous similar message [ 3673.059533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3673.063421] Lustre: Skipped 4 previous similar messages [ 3673.065086] Lustre: Skipped 4 previous similar messages [ 3676.427280] Lustre: server umount lustre-OST0000 complete [ 3677.664065] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3683.735205] Lustre: server umount lustre-MDT0000 complete [ 3684.879464] LustreError: 239195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364622 with bad export cookie 4202237429534457010 [ 3684.882710] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3684.883125] LustreError: 239195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3685.001925] Lustre: server umount lustre-MDT0001 complete [ 3687.707731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3687.861323] LustreError: 243386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3687.868380] LustreError: 243386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3687.888435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3687.890606] Lustre: Skipped 2 previous similar messages [ 3689.135704] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3691.898145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3693.278693] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3695.446343] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3697.100078] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3699.239275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3701.121481] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 3703.358728] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3850.207961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3850.209975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.213052] Lustre: Skipped 2 previous similar messages [ 3850.214679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3850.217862] Lustre: Skipped 6 previous similar messages [ 3854.802485] Lustre: server umount lustre-OST0000 complete [ 3862.355859] Lustre: server umount lustre-MDT0000 complete [ 3863.472209] LustreError: 244122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364801 with bad export cookie 4202237429534457892 [ 3863.475874] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3863.478352] LustreError: 244122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3863.598199] Lustre: server umount lustre-MDT0001 complete [ 3966.494197] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 3967.395498] Key type lgssc unregistered [ 3967.510336] LNet: 246993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3967.513131] LNetError: 246993:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3967.520270] LNet: Removed LNI 192.168.202.104@tcp [ 3967.813095] Key type .llcrypt unregistered [ 3967.814268] Key type ._llcrypt unregistered [ 3975.318183] Key type ._llcrypt registered [ 3975.319379] Key type .llcrypt registered [ 3975.351878] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 3981.715259] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 3982.005881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3982.022063] alg: No test for adler32 (adler32-zlib) [ 3982.862065] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 3982.942464] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 3984.519121] Key type lgssc registered [ 3984.849827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3997.653693] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4000.630928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4000.636053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4001.714027] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4001.723370] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4001.757736] Lustre: lustre-MDT0000: new disk, initializing [ 4001.793747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4001.800861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4003.008606] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4006.794696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4006.830333] Lustre: 251034:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4006.851781] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4006.853735] Lustre: Skipped 1 previous similar message [ 4006.887777] Lustre: lustre-MDT0001: new disk, initializing [ 4006.916811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4006.927498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4006.932486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4008.091808] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4010.235534] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4012.766449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4012.862167] Lustre: lustre-OST0000: new disk, initializing [ 4012.864067] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4012.866631] Lustre: 252664:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4012.899212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4014.329575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4014.333572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4014.373889] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4014.728086] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4019.714958] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4019.791564] Lustre: lustre-OST0001: new disk, initializing [ 4019.793835] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4019.797279] Lustre: 253535:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4019.833987] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4022.018421] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4023.389456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4023.396769] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4023.451764] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4027.986015] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4032.049918] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4037.600740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4037.607221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4037.617681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4039.142382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4039.143328] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4039.156384] Lustre: Skipped 2 previous similar messages [ 4042.377493] Lustre: server umount lustre-MDT0000 complete [ 4043.867572] LustreError: 251026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364981 with bad export cookie 4602032757489960988 [ 4043.872395] LustreError: 251026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4043.872810] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4043.998691] Lustre: server umount lustre-MDT0001 complete [ 4055.691163] Lustre: server umount lustre-OST0000 complete [ 4067.193995] Lustre: server umount lustre-OST0001 complete [ 4073.329078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4077.255797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4080.947766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4085.498555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 4093.066714] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4096.762452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4096.772754] Lustre: lustre-MDT0000: reset Object Index mappings [ 4096.936903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4098.233847] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4101.336285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.343602] Lustre: lustre-MDT0001: reset Object Index mappings [ 4102.783150] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4103.844849] Lustre: 259673:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4106.224099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4106.361400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4106.364866] Lustre: Skipped 1 previous similar message [ 4107.363706] LustreError: 260027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4107.368062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4107.369256] LustreError: 260027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4108.350216] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4111.328599] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4112.432400] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 4113.478500] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4116.814606] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4118.268313] Lustre: 261541:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4122.080810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4122.086332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4122.095705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4122.101237] Lustre: Skipped 3 previous similar messages [ 4125.757698] Lustre: server umount lustre-MDT0000 complete [ 4127.543317] LustreError: 259275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365065 with bad export cookie 4602032757489967764 [ 4127.549436] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4127.550818] LustreError: 259275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4127.686613] Lustre: server umount lustre-MDT0001 complete [ 4139.857598] Lustre: server umount lustre-OST0000 complete [ 4151.655630] Lustre: server umount lustre-OST0001 complete [ 4159.227100] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 01:24:55 (1782365095) [ 4166.477821] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 4169.758527] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4184.508845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4184.617503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4184.631504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4184.674527] Lustre: lustre-MDT0000: new disk, initializing [ 4184.700171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4184.703203] Lustre: Skipped 1 previous similar message [ 4184.708615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4186.345907] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4191.606242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4191.662786] Lustre: 265789:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4191.783733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4193.435712] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4198.101470] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4200.183059] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4203.082167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4203.208309] Lustre: lustre-OST0000: new disk, initializing [ 4203.211433] Lustre: Skipped 1 previous similar message [ 4203.214829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4203.218809] Lustre: Skipped 2 previous similar messages [ 4203.222510] Lustre: 267756:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4205.135116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4205.140859] Lustre: Skipped 1 previous similar message [ 4205.145461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4205.189245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4205.973003] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4210.818157] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4212.558329] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4212.634318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4214.203040] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4214.289361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4215.780898] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4215.781129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4215.782225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4215.782233] Lustre: Skipped 3 previous similar messages [ 4215.796442] Lustre: Skipped 4 previous similar messages [ 4220.897736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4220.899569] Lustre: Skipped 2 previous similar messages [ 4221.261238] Lustre: server umount lustre-OST0000 complete [ 4222.432721] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4222.435068] LustreError: Skipped 1 previous similar message [ 4222.436328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4228.574565] Lustre: server umount lustre-MDT0000 complete [ 4229.720119] LustreError: 266583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365167 with bad export cookie 4602032757489969836 [ 4229.722990] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4229.725381] LustreError: 266583:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4229.834516] Lustre: server umount lustre-MDT0001 complete [ 4235.416174] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4238.821918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4238.972961] LustreError: 270681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4238.988490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4238.990650] Lustre: Skipped 2 previous similar messages [ 4240.348064] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4242.966849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4244.287225] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4245.130987] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 4247.037148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4248.810476] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4251.273356] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4251.329209] Lustre: lustre-OST0001: new disk, initializing [ 4251.331146] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4251.333885] Lustre: 272988:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4252.548358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4252.551751] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4252.589315] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4253.124576] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4258.021824] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4259.549109] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4261.632584] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 4261.638216] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 4263.904458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4263.910141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4263.919987] Lustre: Skipped 1 previous similar message [ 4263.925244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4263.929226] Lustre: Skipped 3 previous similar messages [ 4269.916942] Lustre: server umount lustre-MDT0000 complete [ 4271.009047] LustreError: 270682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4271.013972] LustreError: 270682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4271.045042] LustreError: 270663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365208 with bad export cookie 4602032757489970718 [ 4271.047979] LustreError: 270663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4271.048532] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4271.150128] Lustre: server umount lustre-MDT0001 complete [ 4282.389680] Lustre: server umount lustre-OST0000 complete [ 4293.358116] Lustre: server umount lustre-OST0001 complete [ 4299.373689] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4303.763336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4303.925461] LustreError: 276034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4303.944378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4303.947129] Lustre: Skipped 3 previous similar messages [ 4305.780120] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4308.961618] LustreError: 276035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4309.860673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4311.774742] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4313.197349] Lustre: 277175:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 4316.426402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4319.281077] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4319.723486] LustreError: 277529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4323.354058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4326.068074] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4328.639122] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 4328.651954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 4335.286290] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4336.884347] Lustre: 279055:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 4339.115602] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 4339.122930] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 4343.310604] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 01:28:00 (1782365280) [ 4356.064102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4356.066292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4356.074032] Lustre: Skipped 3 previous similar messages [ 4356.076269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4356.078601] Lustre: Skipped 4 previous similar messages [ 4361.423972] Lustre: server umount lustre-MDT0000 complete [ 4363.314763] LustreError: 276016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365301 with bad export cookie 4602032757489977039 [ 4363.322145] LustreError: 276016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4363.322808] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4364.769507] LustreError: 277552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4364.779340] LustreError: 277552:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4369.585908] Lustre: server umount lustre-MDT0001 complete [ 4377.290996] Lustre: server umount lustre-OST0000 complete [ 4385.157888] Lustre: server umount lustre-OST0001 complete [ 4388.172843] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 4392.387976] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4409.078436] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4413.268075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4413.385056] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4413.404532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4413.445442] Lustre: lustre-MDT0000: new disk, initializing [ 4413.498670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4415.343507] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4420.103569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4420.155815] Lustre: 283286:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4420.273201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4421.871655] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4426.395909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4426.509855] Lustre: 284918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4428.148483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4428.573637] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4432.717988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4432.842366] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4432.844574] Lustre: Skipped 6 previous similar messages [ 4433.950564] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4434.796301] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4439.369777] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4445.520396] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 01:29:42 (1782365382) [ 4449.250322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.252117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4449.254607] Lustre: Skipped 8 previous similar messages [ 4449.257345] Lustre: Skipped 10 previous similar messages [ 4452.503516] Lustre: server umount lustre-MDT0000 complete [ 4454.205276] LustreError: 285789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365392 with bad export cookie 4602032757489984914 [ 4454.209870] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4454.211036] LustreError: 285789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4454.416850] Lustre: server umount lustre-MDT0001 complete [ 4466.440841] Lustre: server umount lustre-OST0000 complete [ 4477.980367] Lustre: server umount lustre-OST0001 complete [ 4480.817783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4480.958870] LustreError: 288035:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4480.965933] LustreError: 288035:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4482.277077] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4485.078380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4486.472757] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4488.790995] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4490.575895] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4492.831016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4494.755358] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4496.993565] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4513.387705] loop1: detected capacity change from 0 to 204800000 [ 4513.468774] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4513.472969] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4513.478165] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4515.273075] loop1: detected capacity change from 0 to 204800000 [ 4517.268659] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4517.302556] LustreError: 288023:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs22658: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [ 4517.307381] LustreError: 288023:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs22658 [ 4517.310436] LustreError: 288023:0:(mgs_handler.c:517:mgs_target_reg()) fs22658-MDT0001: Error checking target index: rc = -13 [ 4517.314214] LustreError: fs22658-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4517.318844] LustreError: 291771:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 4517.320984] LustreError: 291771:0:(tgt_mount.c:2129:server_put_super()) no obd fs22658-MDT0001 [ 4517.323493] LustreError: 291771:0:(tgt_mount.c:116:server_deregister_mount()) fs22658-MDT0001 not registered [ 4517.335685] Lustre: server umount fs22658-MDT0001 complete [ 4517.337215] LustreError: 291771:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4520.207561] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4520.242941] Lustre: Setting parameter fs22658-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs22658-MDT0001 [ 4520.250173] Lustre: Skipped 2 previous similar messages [ 4520.266851] Lustre: srv-fs22658-MDT0001: No data found on store. Initialize space. [ 4520.269189] Lustre: Skipped 4 previous similar messages [ 4520.302933] Lustre: fs22658-MDT0001: new disk, initializing [ 4520.304553] Lustre: Skipped 3 previous similar messages [ 4520.335693] Lustre: srv-fs22658-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 4520.339375] Lustre: 292236:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22658-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4520.342456] LustreError: 292236:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22658-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4521.789975] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4524.742657] Lustre: server umount fs22658-MDT0001 complete [ 4529.228455] loop1: detected capacity change from 0 to 204800000 [ 4531.127284] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4531.244393] Lustre: 293803:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22658-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4531.248995] LustreError: 293803:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22658-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4532.559412] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4534.131747] Lustre: server umount fs22658-MDT0001 complete [ 4537.313669] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4537.314566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4537.317027] Lustre: Skipped 1 previous similar message [ 4537.320139] Lustre: Skipped 2 previous similar messages [ 4543.378718] Lustre: server umount lustre-MDT0000 complete [ 4545.983844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4546.022916] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4546.082437] LustreError: 289864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4546.087477] LustreError: 289864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 4547.366957] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4549.818186] loop1: detected capacity change from 0 to 204800000 [ 4551.142456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4551.636549] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4551.752993] Lustre: 296245:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs22658-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4551.755994] LustreError: 296245:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs22658-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4553.013285] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4554.545369] Lustre: server umount fs22658-MDT0001 complete [ 4561.379521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4567.120537] Lustre: server umount lustre-OST0000 complete [ 4574.488122] Lustre: server umount lustre-MDT0000 complete [ 4575.647644] LustreError: 288015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365513 with bad export cookie 4602032757489987588 [ 4575.777601] Lustre: server umount lustre-MDT0001 complete [ 4579.253766] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 4580.191951] Key type lgssc unregistered [ 4580.307313] LNet: 298584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4580.309304] LNetError: 298584:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4580.324242] LNet: Removed LNI 192.168.202.104@tcp [ 4580.601115] Key type .llcrypt unregistered [ 4580.602153] Key type ._llcrypt unregistered [ 4588.220514] Key type ._llcrypt registered [ 4588.222108] Key type .llcrypt registered [ 4588.260868] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4588.674255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4588.680493] alg: No test for adler32 (adler32-zlib) [ 4589.525614] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 4589.616555] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 4591.191180] Key type lgssc registered [ 4591.549658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4594.077630] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4594.086378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4595.236897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4596.535086] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4599.325200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4599.479819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4600.835819] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4603.168946] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4604.990244] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4607.207622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4607.335275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4609.232971] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4609.380474] LustreError: 301657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4609.385556] LustreError: 301657:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4611.592752] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4614.628127] LustreError: 301658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4619.747153] LustreError: 301658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4622.077489] LustreError: 301657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4622.082035] LustreError: 301657:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4627.192818] LustreError: 301656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4627.197907] LustreError: 301656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4628.206129] loop1: detected capacity change from 0 to 204800000 [ 4628.277491] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4628.281175] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4628.284875] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4630.061947] loop1: detected capacity change from 0 to 204800000 [ 4632.061968] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4632.122405] Lustre: fs11967-OST0000: new disk, initializing [ 4632.124105] Lustre: srv-fs11967-OST0000: No data found on store. Initialize space. [ 4632.126692] Lustre: 303582:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4632.152832] Lustre: fs11967-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4634.053752] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4636.132312] LustreError: 301657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4636.140737] LustreError: 301657:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4637.476688] Lustre: server umount fs11967-OST0000 complete [ 4641.562037] loop1: detected capacity change from 0 to 204800000 [ 4643.510864] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4643.567331] Lustre: fs11967-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4645.413861] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4648.260395] loop2: detected capacity change from 0 to 204800000 [ 4648.360893] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4648.364471] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4648.368672] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 4650.091682] loop2: detected capacity change from 0 to 204800000 [ 4651.881541] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4651.912940] LustreError: fs11967-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [ 4651.916760] LustreError: fs11967-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [ 4651.919345] LustreError: 306620:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [ 4651.921443] LustreError: 306620:0:(tgt_mount.c:2129:server_put_super()) no obd fs11967-OST0001 [ 4651.923193] LustreError: 306620:0:(tgt_mount.c:116:server_deregister_mount()) fs11967-OST0001 not registered [ 4651.932569] Lustre: server umount fs11967-OST0001 complete [ 4651.933900] LustreError: 306620:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [ 4652.791041] LustreError: 301656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.798532] LustreError: 301656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 4655.178435] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4655.220823] Lustre: fs11967-OST0001: new disk, initializing [ 4655.222393] Lustre: srv-fs11967-OST0001: No data found on store. Initialize space. [ 4655.224773] Lustre: 307170:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4655.244250] Lustre: fs11967-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4657.118428] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4660.520103] Lustre: server umount fs11967-OST0000 complete [ 4663.662204] Lustre: server umount fs11967-OST0001 complete [ 4667.665272] loop1: detected capacity change from 0 to 204800000 [ 4669.658277] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4671.714495] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4673.785991] loop2: detected capacity change from 0 to 204800000 [ 4675.566742] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4675.623243] Lustre: fs11967-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4675.625144] Lustre: Skipped 1 previous similar message [ 4677.497119] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4680.297344] Lustre: server umount fs11967-OST0000 complete [ 4683.289516] Lustre: server umount fs11967-OST0001 complete [ 4687.331394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4687.331394] LustreError: 308934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4687.331445] LustreError: 308934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [ 4687.332779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4687.332971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4687.335707] Lustre: Skipped 1 previous similar message [ 4692.448938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4692.451901] Lustre: Skipped 2 previous similar messages [ 4697.568843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4697.570162] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [ 4697.570569] Lustre: Skipped 1 previous similar message [ 4701.151133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4701.198507] Lustre: server umount lustre-OST0000 complete [ 4702.688322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4702.691925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4707.296149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4707.299767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4707.306074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4708.508592] Lustre: server umount lustre-MDT0000 complete [ 4709.738124] LustreError: 301679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365647 with bad export cookie 14944473569285960674 [ 4709.742472] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4709.743121] LustreError: 301679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4709.866559] Lustre: server umount lustre-MDT0001 complete [ 4713.723639] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 4714.814631] Key type lgssc unregistered [ 4714.942267] LNet: 313083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4714.944307] LNetError: 313083:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4714.955294] LNet: Removed LNI 192.168.202.104@tcp [ 4715.272103] Key type .llcrypt unregistered [ 4715.273194] Key type ._llcrypt unregistered [ 4726.032535] Key type ._llcrypt registered [ 4726.033589] Key type .llcrypt registered [ 4726.066681] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 4732.181721] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4732.588212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4732.627518] alg: No test for adler32 (adler32-zlib) [ 4733.469643] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 4733.551815] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 4735.127107] Key type lgssc registered [ 4735.449934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4747.293548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4747.298638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4748.386859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4748.399652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4748.438136] Lustre: lustre-MDT0000: new disk, initializing [ 4748.461460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4748.467800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4749.910936] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4754.019072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4754.061184] Lustre: 316911:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4754.106380] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4754.108129] Lustre: Skipped 1 previous similar message [ 4754.142130] Lustre: lustre-MDT0001: new disk, initializing [ 4754.177818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4754.188886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4754.192541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4755.562302] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4759.227511] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4761.142793] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4763.407169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4763.526741] Lustre: lustre-OST0000: new disk, initializing [ 4763.528602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4763.531162] Lustre: 318881:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4763.557926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4763.744742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4763.748220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4763.789509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4765.395316] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4769.306740] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4769.978313] Lustre: Mounted lustre-client [ 4770.301141] Lustre: Unmounted lustre-client [ 4774.368538] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4774.368809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4774.370802] LustreError: Skipped 1 previous similar message [ 4774.371456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4774.377560] Lustre: Skipped 1 previous similar message [ 4775.396603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4775.399320] Lustre: Skipped 2 previous similar messages [ 4779.175497] Lustre: server umount lustre-OST0000 complete [ 4785.120476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4785.124191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4785.128733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4790.753112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4790.757364] Lustre: Skipped 2 previous similar messages [ 4795.359091] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4795.428215] Lustre: server umount lustre-MDT0000 complete [ 4796.687860] LustreError: 316905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365734 with bad export cookie 10006866486499613802 [ 4796.693452] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4796.694855] LustreError: 316905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4796.846742] Lustre: server umount lustre-MDT0001 complete [ 4800.263434] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 4801.305058] Key type lgssc unregistered [ 4826.015523] LNet: 321043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4826.020322] LNetError: 321043:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4900.773281] LNet: Removed LNI 192.168.202.104@tcp [ 4901.047713] Key type .llcrypt unregistered [ 4901.048809] Key type ._llcrypt unregistered [ 4906.480966] Key type ._llcrypt registered [ 4906.482060] Key type .llcrypt registered [ 4906.519343] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 4913.870003] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4914.188052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4914.218060] alg: No test for adler32 (adler32-zlib) [ 4915.072441] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 4915.158091] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 4916.751142] Key type lgssc registered [ 4917.236462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4931.558982] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 01:37:48 (1782365868) [ 4934.854326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4934.865793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4935.973297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4935.983346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4936.023603] Lustre: lustre-MDT0000: new disk, initializing [ 4936.043810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4936.049274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4937.574659] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4942.511556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4942.573321] Lustre: 324247:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4942.615426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4942.619351] Lustre: Skipped 1 previous similar message [ 4942.668784] Lustre: lustre-MDT0001: new disk, initializing [ 4942.710588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4942.723714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4942.726845] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4944.404368] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4948.553860] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4950.747087] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4953.747265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4953.890749] Lustre: lustre-OST0000: new disk, initializing [ 4953.893608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4953.897849] Lustre: 326213:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4953.936843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4955.958300] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 4958.025169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4958.031400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4958.074293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4960.088929] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4968.420558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4968.421633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4968.426107] LustreError: Skipped 1 previous similar message [ 4968.436585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4968.442490] Lustre: Skipped 1 previous similar message [ 4972.121452] Lustre: server umount lustre-OST0000 complete [ 4973.538083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4973.538084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4973.538107] Lustre: Skipped 1 previous similar message [ 4973.538995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4978.657069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4978.661395] Lustre: Skipped 2 previous similar messages [ 4979.493604] Lustre: server umount lustre-MDT0000 complete [ 4981.204898] LustreError: 325040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365919 with bad export cookie 6339219219560663515 [ 4981.212492] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4981.214349] LustreError: 325040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4981.363848] Lustre: server umount lustre-MDT0001 complete [ 4985.404478] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 4986.710339] Key type lgssc unregistered [ 4986.830339] LNet: 328123:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4986.832924] LNetError: 328123:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4986.842298] LNet: Removed LNI 192.168.202.104@tcp [ 4987.216113] Key type .llcrypt unregistered [ 4987.217688] Key type ._llcrypt unregistered [ 4996.559022] Key type ._llcrypt registered [ 4996.560109] Key type .llcrypt registered [ 4996.599075] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 4997.006838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4997.013483] alg: No test for adler32 (adler32-zlib) [ 4997.894182] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 4998.004032] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 4999.583225] Key type lgssc registered [ 5000.107514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5003.916901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5003.929972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5005.128267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5006.968893] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5011.068680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5011.218367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5012.797054] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5015.813417] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5018.162762] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5020.883073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5021.065529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5023.464069] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5026.428774] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5034.919562] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5035.642859] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5040.609266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5040.610760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5040.613521] Lustre: Skipped 2 previous similar messages [ 5040.615499] Lustre: Skipped 1 previous similar message [ 5041.922689] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.4@tcp (stopping) [ 5041.926993] Lustre: Skipped 1 previous similar message [ 5043.414913] Lustre: server umount lustre-MDT0000 complete [ 5044.606278] LustreError: 329354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365982 with bad export cookie 3659020226383413094 [ 5044.608764] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5044.611923] LustreError: 329354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5045.728369] LustreError: 329368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5045.728776] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5045.735490] LustreError: 329368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5045.742406] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5047.029221] LustreError: 329370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5049.759884] LustreError: 329368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5049.760052] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5049.765481] Lustre: Skipped 1 previous similar message [ 5050.794574] Lustre: server umount lustre-MDT0001 complete [ 5057.269210] Lustre: lustre-OST0000: Not available for connect from 192.168.202.4@tcp (stopping) [ 5057.272798] Lustre: Skipped 1 previous similar message [ 5066.207162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5066.285157] Lustre: server umount lustre-OST0000 complete [ 5069.830380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5071.257779] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5072.214439] Lustre: 333548:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.104@tcp [ 5073.046464] Lustre: server umount MGS complete [ 5075.985518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5076.148177] LustreError: 334120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5076.172190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5077.647408] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5080.870223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5081.049675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5083.079241] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5086.515309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5088.622837] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5091.624465] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5093.316694] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 5095.358342] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5103.321102] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec [ 5106.144585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5106.149590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5106.158449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5106.162215] Lustre: Skipped 2 previous similar messages [ 5107.174360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5110.809105] Lustre: server umount lustre-OST0000 complete [ 5116.895846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5116.897922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5116.901023] Lustre: Skipped 1 previous similar message [ 5118.706369] Lustre: server umount lustre-MDT0000 complete [ 5120.349452] LustreError: 334100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782366058 with bad export cookie 3659020226383413920 [ 5120.354923] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5120.359062] LustreError: 334100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5120.508749] Lustre: server umount lustre-MDT0001 complete [ 5124.691996] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5126.049328] Key type lgssc unregistered [ 5126.197671] LNet: 337554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5126.202867] LNetError: 337554:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5126.211744] LNet: Removed LNI 192.168.202.104@tcp [ 5126.584360] Key type .llcrypt unregistered [ 5126.586720] Key type ._llcrypt unregistered [ 5134.651116] Key type ._llcrypt registered [ 5134.652551] Key type .llcrypt registered [ 5134.687219] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 5142.161027] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 5142.514792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5142.524572] alg: No test for adler32 (adler32-zlib) [ 5143.381602] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 5143.479721] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 5145.087192] Key type lgssc registered [ 5145.480644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5161.293785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5161.305751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5162.425532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5162.442896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5162.502871] Lustre: lustre-MDT0000: new disk, initializing [ 5162.542995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5162.554630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5164.346824] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5169.695330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5169.776587] Lustre: 340689:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5169.826262] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5169.830283] Lustre: Skipped 1 previous similar message [ 5169.881348] Lustre: lustre-MDT0001: new disk, initializing [ 5169.929275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5169.947886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5169.953402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5171.677509] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5176.350504] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5178.463897] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5181.348822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5181.463975] Lustre: lustre-OST0000: new disk, initializing [ 5181.465506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5181.467923] Lustre: 342657:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5181.498622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5183.702087] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5188.102390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5188.110046] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5188.171982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5188.190450] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5190.049200] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5190.172791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5192.248332] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5192.336667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5198.306337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5198.308612] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5198.311047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5198.311786] Lustre: Skipped 1 previous similar message [ 5199.408815] Lustre: server umount lustre-OST0000 complete [ 5200.863828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5200.866470] LustreError: Skipped 1 previous similar message [ 5200.867985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5200.872409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5200.874325] Lustre: Skipped 1 previous similar message [ 5203.425977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5203.427776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5208.544979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5208.548591] Lustre: Skipped 2 previous similar messages [ 5213.665316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5213.670464] Lustre: Skipped 1 previous similar message [ 5215.199126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5215.268936] Lustre: server umount lustre-MDT0000 complete [ 5216.883759] LustreError: 340680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782366154 with bad export cookie 13173591816666603001 [ 5216.889777] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5216.890891] LustreError: 340680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5217.019097] Lustre: server umount lustre-MDT0001 complete [ 5220.250860] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 01:42:36 (1782366156) [ 5225.827405] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 5228.768867] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 5241.864507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5241.999883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5242.018623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5242.075495] Lustre: lustre-MDT0000: new disk, initializing [ 5242.108919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5242.119344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5243.936561] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5249.406622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5249.485100] Lustre: 347404:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5249.527366] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5249.531247] Lustre: Skipped 1 previous similar message [ 5249.578069] Lustre: lustre-MDT0001: new disk, initializing [ 5249.611212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5249.625856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5249.630254] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5251.302080] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5255.612985] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5257.957239] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5260.755440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5260.912491] Lustre: lustre-OST0000: new disk, initializing [ 5260.915086] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5260.918806] Lustre: 349373:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5260.955147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5263.454873] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5268.265426] Lustre: DEBUG MARKER: oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5268.480568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5268.485146] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5268.525295] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5283.808564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5283.809295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5283.810938] LustreError: Skipped 1 previous similar message [ 5283.816898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5283.819677] Lustre: Skipped 1 previous similar message [ 5288.907732] Lustre: server umount lustre-OST0000 complete [ 5290.463890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5290.466724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5290.471535] Lustre: Skipped 1 previous similar message [ 5296.210175] Lustre: server umount lustre-MDT0000 complete [ 5297.416610] LustreError: 347399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782366235 with bad export cookie 13173591816666603869 [ 5297.420469] LustreError: 347399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5297.421048] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5297.546113] Lustre: server umount lustre-MDT0001 complete [ 5300.819776] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5301.714619] Key type lgssc unregistered [ 5301.840312] LNet: 351522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5301.842539] LNetError: 351522:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5301.849269] LNet: Removed LNI 192.168.202.104@tcp [ 5302.146724] Key type .llcrypt unregistered [ 5302.147755] Key type ._llcrypt unregistered [ 5333.714394] Key type ._llcrypt registered [ 5333.715608] Key type .llcrypt registered [ 5333.754598] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5333.862486] Key type .llcrypt unregistered [ 5333.863977] Key type ._llcrypt unregistered [ 5367.623467] Key type ._llcrypt registered [ 5367.624779] Key type .llcrypt registered [ 5367.671342] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5367.788189] Key type .llcrypt unregistered [ 5367.789501] Key type ._llcrypt unregistered [ 5379.514359] Key type ._llcrypt registered [ 5379.515572] Key type .llcrypt registered [ 5379.559541] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5379.663273] Key type .llcrypt unregistered [ 5379.664577] Key type ._llcrypt unregistered [ 5390.454694] Key type ._llcrypt registered [ 5390.456391] Key type .llcrypt registered [ 5390.492530] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5390.594333] Key type .llcrypt unregistered [ 5390.595412] Key type ._llcrypt unregistered [ 5418.616404] Key type ._llcrypt registered [ 5418.617455] Key type .llcrypt registered [ 5418.651837] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5418.739470] Key type .llcrypt unregistered [ 5418.740751] Key type ._llcrypt unregistered [ 5429.173484] Key type ._llcrypt registered [ 5429.174464] Key type .llcrypt registered [ 5429.207936] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5429.301350] Key type .llcrypt unregistered [ 5429.302362] Key type ._llcrypt unregistered [ 5441.016355] Key type ._llcrypt registered [ 5441.017900] Key type .llcrypt registered [ 5441.051439] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5441.141183] Key type .llcrypt unregistered [ 5441.142195] Key type ._llcrypt unregistered [ 5452.011322] Key type ._llcrypt registered [ 5452.012411] Key type .llcrypt registered [ 5452.050355] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5452.131379] Key type .llcrypt unregistered [ 5452.132392] Key type ._llcrypt unregistered [ 5463.920065] Key type ._llcrypt registered [ 5463.921579] Key type .llcrypt registered [ 5463.975817] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5464.081216] Key type .llcrypt unregistered [ 5464.082274] Key type ._llcrypt unregistered [ 5475.323630] Key type ._llcrypt registered [ 5475.324679] Key type .llcrypt registered [ 5475.358138] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5475.444196] Key type .llcrypt unregistered [ 5475.445383] Key type ._llcrypt unregistered [ 5496.577248] Key type ._llcrypt registered [ 5496.578711] Key type .llcrypt registered [ 5496.618814] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5496.720304] Key type .llcrypt unregistered [ 5496.721379] Key type ._llcrypt unregistered [ 5509.056108] Key type ._llcrypt registered [ 5509.057165] Key type .llcrypt registered [ 5509.099248] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5509.205226] Key type .llcrypt unregistered [ 5509.206463] Key type ._llcrypt unregistered [ 5519.637592] Key type ._llcrypt registered [ 5519.638517] Key type .llcrypt registered [ 5519.673987] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5519.776448] Key type .llcrypt unregistered [ 5519.778060] Key type ._llcrypt unregistered [ 5540.024147] Key type ._llcrypt registered [ 5540.025362] Key type .llcrypt registered [ 5540.064839] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 5546.958694] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 5547.380047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5547.409109] alg: No test for adler32 (adler32-zlib) [ 5548.295179] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 5548.398783] LNet: Added LNI 192.168.202.104@tcp [8/256/0/180] [ 5549.999257] Key type lgssc registered [ 5550.409334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5563.836913] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5563.846806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5564.956449] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [ 5564.971812] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [ 5565.020614] Lustre: tst250fs-MDT0000: new disk, initializing [ 5565.054070] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5565.065259] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5566.644098] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5571.208106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5571.257482] Lustre: 371423:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5571.287663] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [ 5571.290008] Lustre: Skipped 1 previous similar message [ 5571.325347] Lustre: tst250fs-MDT0001: new disk, initializing [ 5571.362149] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5571.374320] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5571.377393] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5572.961592] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5575.185829] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 5577.301633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5577.424781] Lustre: tst250fs-OST0000: new disk, initializing [ 5577.427194] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [ 5577.430645] Lustre: 373057:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5577.465245] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5579.343361] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5580.156118] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5580.159119] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5580.206500] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5584.476374] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5584.571142] Lustre: tst250fs-OST0001: new disk, initializing [ 5584.573994] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [ 5584.578539] Lustre: 373930:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5584.625223] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5587.218809] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_default_debug -1 all [ 5587.868608] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5587.874556] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5587.930142] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5590.097350] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [ 5602.272259] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5602.276233] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5602.282427] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5603.301425] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [ 5603.302568] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.305386] Lustre: Skipped 1 previous similar message [ 5603.313133] Lustre: Skipped 2 previous similar messages [ 5607.222953] Lustre: server umount tst250fs-MDT0000 complete [ 5608.874288] LustreError: 371460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782366546 with bad export cookie 5728564851062324973 [ 5608.881463] LustreError: MGC192.168.202.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5608.884857] LustreError: 371460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5609.058770] Lustre: server umount tst250fs-MDT0001 complete [ 5620.752772] Lustre: server umount tst250fs-OST0000 complete [ 5632.623253] Lustre: server umount tst250fs-OST0001 complete [ 5638.017786] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 5640.940713] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 5652.222264] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 01:49:48 (1782366588) [ 5653.064696] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [ 5659.850086] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing set_hostid [ 5664.844291] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing load_modules_local [ 5678.572777] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 5509 sec ========== 01:50:15 (1782366615) [ 5679.338488] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:50:15 (1782366615) === [ 5680.637432] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:50:17 (1782366617) === [ 5695.849618] Lustre: DEBUG MARKER: oleg204-server.virtnet: executing unload_modules_local [ 5696.948089] Key type lgssc unregistered [ 5697.084358] LNet: 383038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5697.087381] LNetError: 383038:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5697.099305] LNet: Removed LNI 192.168.202.104@tcp [ 5697.438100] Key type .llcrypt unregistered [ 5697.439558] Key type ._llcrypt unregistered