[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 493640699 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002354] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.006561] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008014] pid_max: default: 32768 minimum: 301 [ 0.009219] LSM: Security Framework initializing [ 0.010059] Yama: becoming mindful. [ 0.011042] SELinux: Initializing. [ 0.012088] *** VALIDATE selinux *** [ 0.020694] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025622] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026181] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027121] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028130] *** VALIDATE tmpfs *** [ 0.029500] *** VALIDATE proc *** [ 0.031223] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.034091] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036039] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037031] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041536] debug: unmapping init [mem 0xffffffffad059000-0xffffffffad060fff] [ 0.043863] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044676] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045028] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048011] ... value mask: 0000ffffffffffff [ 0.049012] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052319] rcu: Hierarchical SRCU implementation. [ 0.054479] smp: Bringing up secondary CPUs ... [ 0.055629] x86: Booting SMP configuration: [ 0.056037] .... node #0, CPUs: #1 #2 #3 [ 0.060092] smp: Brought up 1 node, 4 CPUs [ 0.062030] smpboot: Max logical packages: 1 [ 0.063026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.236127] node 0 deferred pages initialised in 170ms [ 0.240229] devtmpfs: initialized [ 0.241180] x86/mm: Memory block size: 128MB [ 0.243561] gcov: version magic: 0x41383552 [ 0.244579] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.249162] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.251462] pinctrl core: initialized pinctrl subsystem [ 0.253229] [ 0.253758] ************************************************************* [ 0.256020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258016] ** ** [ 0.260018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.262018] ** ** [ 0.264019] ** This means that this kernel is built to expose internal ** [ 0.265016] ** IOMMU data structures, which may compromise security on ** [ 0.267018] ** your system. ** [ 0.269018] ** ** [ 0.272020] ** If you see this message and you are not debugging the ** [ 0.274018] ** kernel, report this immediately to your vendor! ** [ 0.276019] ** ** [ 0.278022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.280021] ************************************************************* [ 0.281000] NET: Registered protocol family 16 [ 0.283542] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.287091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.290144] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.295049] cpuidle: using governor menu [ 0.297000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.301565] PCI: Using configuration type 1 for base access [ 0.303197] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.311200] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.314034] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.319005] cryptd: max_cpu_qlen set to 1000 [ 0.322294] ACPI: Added _OSI(Module Device) [ 0.323016] ACPI: Added _OSI(Processor Device) [ 0.324014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.325030] ACPI: Added _OSI(Processor Aggregator Device) [ 0.330530] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.337009] ACPI: Interpreter enabled [ 0.338060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.340013] ACPI: Using IOAPIC for interrupt routing [ 0.342207] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.343322] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.359592] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.363071] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.368037] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.374102] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.381409] acpiphp: Slot [2] registered [ 0.383156] acpiphp: Slot [5] registered [ 0.386142] acpiphp: Slot [6] registered [ 0.388150] acpiphp: Slot [7] registered [ 0.391146] acpiphp: Slot [8] registered [ 0.393161] acpiphp: Slot [9] registered [ 0.396168] acpiphp: Slot [10] registered [ 0.398178] acpiphp: Slot [3] registered [ 0.401198] acpiphp: Slot [4] registered [ 0.402115] acpiphp: Slot [11] registered [ 0.404194] acpiphp: Slot [12] registered [ 0.406175] acpiphp: Slot [13] registered [ 0.407123] acpiphp: Slot [14] registered [ 0.409184] acpiphp: Slot [15] registered [ 0.411169] acpiphp: Slot [16] registered [ 0.413175] acpiphp: Slot [17] registered [ 0.415181] acpiphp: Slot [18] registered [ 0.417153] acpiphp: Slot [19] registered [ 0.418190] acpiphp: Slot [20] registered [ 0.420160] acpiphp: Slot [21] registered [ 0.422163] acpiphp: Slot [22] registered [ 0.424137] acpiphp: Slot [23] registered [ 0.425127] acpiphp: Slot [24] registered [ 0.427156] acpiphp: Slot [25] registered [ 0.429216] acpiphp: Slot [26] registered [ 0.430154] acpiphp: Slot [27] registered [ 0.432157] acpiphp: Slot [28] registered [ 0.434186] acpiphp: Slot [29] registered [ 0.437204] acpiphp: Slot [30] registered [ 0.439182] acpiphp: Slot [31] registered [ 0.441075] PCI host bridge to bus 0000:00 [ 0.443036] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.446024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.451027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.453025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.456024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.460025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.462225] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.465954] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.469246] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.479761] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.485106] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.487019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.490020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.493020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.495460] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.498807] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.501063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.505016] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.511027] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.526037] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.533025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.540280] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.549017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.556017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.572015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.582969] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.590025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.597028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.619024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.635073] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.644021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.653023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.671040] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.682359] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.692023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.699023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.722021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.733736] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.741024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.750018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.766017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.775253] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.788075] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.798027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.823028] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.838799] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.842392] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.845847] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.848451] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.851236] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.856147] iommu: Default domain type: Passthrough [ 0.858442] SCSI subsystem initialized [ 0.860138] ACPI: bus type USB registered [ 0.862099] usbcore: registered new interface driver usbfs [ 0.864141] usbcore: registered new interface driver hub [ 0.866076] usbcore: registered new device driver usb [ 0.868194] pps_core: LinuxPPS API ver. 1 registered [ 0.871015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.873053] PTP clock support registered [ 0.875122] EDAC MC: Ver: 3.0.0 [ 0.876599] PCI: Using ACPI for IRQ routing [ 0.879176] NetLabel: Initializing [ 0.881021] NetLabel: domain hash size = 128 [ 0.883020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.885137] NetLabel: unlabeled traffic allowed by default [ 0.888209] vgaarb: loaded [ 0.890274] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.891014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.896306] clocksource: Switched to clocksource kvm-clock [ 1.015572] VFS: Disk quotas dquot_6.6.0 [ 1.017365] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.020416] *** VALIDATE ramfs *** [ 1.021565] *** VALIDATE hugetlbfs *** [ 1.023289] pnp: PnP ACPI init [ 1.025908] pnp: PnP ACPI: found 6 devices [ 1.048405] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.053262] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.056577] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.059800] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.063197] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.066394] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.071216] NET: Registered protocol family 2 [ 1.075052] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.081109] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.085480] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.091614] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.095509] TCP: Hash tables configured (established 65536 bind 65536) [ 1.098807] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.101779] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.104548] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.107952] NET: Registered protocol family 1 [ 1.111202] RPC: Registered named UNIX socket transport module. [ 1.114330] RPC: Registered udp transport module. [ 1.116842] RPC: Registered tcp transport module. [ 1.119150] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.122144] NET: Registered protocol family 44 [ 1.124054] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.126642] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.128875] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.131123] PCI: CLS 0 bytes, default 64 [ 1.133403] Unpacking initramfs... [ 2.491948] debug: unmapping init [mem 0xffff89fc7cc54000-0xffff89fc7ffbffff] [ 2.496235] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.499567] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.502373] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.001653] Initialise system trusted keyrings [ 3.003390] Key type blacklist registered [ 3.005328] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.015131] zbud: loaded [ 3.018584] *** VALIDATE nfs *** [ 3.019978] *** VALIDATE nfs4 *** [ 3.021774] pstore: using deflate compression [ 3.025499] Platform Keyring initialized [ 3.130116] NET: Registered protocol family 38 [ 3.131946] Key type asymmetric registered [ 3.133529] Asymmetric key parser 'x509' registered [ 3.135368] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.138562] io scheduler mq-deadline registered [ 3.140392] io scheduler kyber registered [ 3.142358] io scheduler bfq registered [ 3.147611] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.151685] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.153949] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.157548] ACPI: Power Button [PWRF] [ 3.163704] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.170559] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.190230] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.197257] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.210162] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.238127] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.265682] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.272167] Non-volatile memory driver v1.3 [ 3.274707] Linux agpgart interface v0.103 [ 3.304809] virtio_blk virtio1: [vda] 147000 512-byte logical blocks (75.3 MB/71.8 MiB) [ 3.308208] vda: detected capacity change from 0 to 75264000 [ 3.321912] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.325292] vdb: detected capacity change from 0 to 1073741824 [ 3.337968] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.340885] vdc: detected capacity change from 0 to 2621440000 [ 3.355285] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.358927] vdd: detected capacity change from 0 to 2621440000 [ 3.380837] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.383932] vde: detected capacity change from 0 to 4294967296 [ 3.401293] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.404790] vdf: detected capacity change from 0 to 4294967296 [ 3.412152] libphy: Fixed MDIO Bus: probed [ 3.418620] usbcore: registered new interface driver usbserial_generic [ 3.421163] usbserial: USB Serial support registered for generic [ 3.423587] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.427731] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.429588] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.432171] mousedev: PS/2 mouse device common for all mice [ 3.435577] rtc_cmos 00:05: RTC can wake from S4 [ 3.437909] rtc_cmos 00:05: registered as rtc0 [ 3.440597] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.441739] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.443109] intel_pstate: CPU model not supported [ 3.450233] hid: raw HID events driver (C) Jiri Kosina [ 3.452032] usbcore: registered new interface driver usbhid [ 3.453434] usbhid: USB HID core driver [ 3.454578] drop_monitor: Initializing network drop monitor service [ 3.456556] Initializing XFRM netlink socket [ 3.457180] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.458388] NET: Registered protocol family 10 [ 3.462931] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.463950] Segment Routing with IPv6 [ 3.466859] NET: Registered protocol family 17 [ 3.468603] mpls_gso: MPLS GSO support [ 3.474264] RAS: Correctable Errors collector initialized. [ 3.475872] AVX version of gcm_enc/dec engaged. [ 3.477086] AES CTR mode by8 optimization enabled [ 3.550680] sched_clock: Marking stable (3550602392, 0)->(4483159909, -932557517) [ 3.554237] registered taskstats version 1 [ 3.556370] Loading compiled-in X.509 certificates [ 3.558833] zswap: loaded using pool lzo/zbud [ 3.587756] Key type big_key registered [ 3.605372] Key type encrypted registered [ 3.607712] ima: No TPM chip found, activating TPM-bypass! [ 3.610329] ima: Allocated hash algorithm: sha1 [ 3.612353] ima: No architecture policies found [ 3.614287] evm: Initialising EVM extended attributes: [ 3.616476] evm: security.selinux [ 3.618079] evm: security.ima [ 3.619272] evm: security.capability [ 3.621460] evm: HMAC attrs: 0x1 [ 3.624302] rtc_cmos 00:05: setting system clock to 2026-08-28 01:31:18 UTC (1787880678) [ 3.632868] debug: unmapping init [mem 0xffffffffae003000-0xffffffffae1fffff] [ 3.635702] debug: unmapping init [mem 0xffffffffacd82000-0xffffffffad058fff] [ 3.644104] Write protecting the kernel read-only data: 28672k [ 3.647612] debug: unmapping init [mem 0xffffffffab403000-0xffffffffab5fffff] [ 3.650625] debug: unmapping init [mem 0xffffffffabd14000-0xffffffffabdfffff] [ 3.690517] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.700523] systemd[1]: Detected virtualization kvm. [ 3.703245] systemd[1]: Detected architecture x86-64. [ 3.705956] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.732604] systemd[1]: No hostname configured. [ 3.734595] systemd[1]: Set hostname to . [ 3.736687] random: systemd: uninitialized urandom read (16 bytes read) [ 3.739401] systemd[1]: Initializing machine ID from random generator. [ 3.793583] random: ln: uninitialized urandom read (6 bytes read) [ 3.886619] random: systemd: uninitialized urandom read (16 bytes read) [ 3.890745] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.896852] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.901792] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.646562] device-mapper: uevent: version 1.0.3 [ 4.649215] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.495635] virtio_net virtio0 ens2: renamed from eth0 [ 5.525145] random: fast init done [ 5.567598] scsi host0: ata_piix [ 5.639266] scsi host1: ata_piix [ 5.641452] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.644568] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.471704] dracut-initqueue[596]: RTNETLINK answers: File exists [ 10.127142] random: crng init done [ 10.130371] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.785514] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.115180] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.391097] SELinux: Disabled at runtime. [ 12.458179] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.466798] systemd[1]: Detected virtualization kvm. [ 12.468652] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.092959] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.096764] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.102809] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.108334] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.112635] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.120393] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.130600] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ 13.195040] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.677490] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.042224] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.050557] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.265573] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.277354] EDAC sbridge: Ver: 1.1.2 [ 16.326183] Key type dns_resolver registered [ 16.692211] NFS: Registering the id_resolver key type [ 16.694623] Key type id_resolver registered [ 16.696017] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg431-server login: [ 66.186919] libcfs: loading out-of-tree module taints kernel. [ 66.212792] Key type ._llcrypt registered [ 66.214395] Key type .llcrypt registered [ 66.372073] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 88.300152] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 90.610174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 90.652493] alg: No test for adler32 (adler32-zlib) [ 92.532821] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 93.886193] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 95.704478] Key type lgssc registered [ 97.728567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 119.178989] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 156.165123] hrtimer: interrupt took 21114812 ns [ 168.702025] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 183.426762] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 183.472132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 184.715714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 184.769743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 184.878364] Lustre: lustre-MDT0000: new disk, initializing [ 185.021648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 185.063570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 189.946882] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.320541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 203.396359] Lustre: 6531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 203.445497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 203.449265] Lustre: Skipped 1 previous similar message [ 203.522435] Lustre: lustre-MDT0001: new disk, initializing [ 203.576959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.610841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 203.626163] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 208.214923] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.822745] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 222.937160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.236335] Lustre: lustre-OST0000: new disk, initializing [ 223.241720] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 223.249598] Lustre: 8468:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 223.337245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.183026] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.452752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 231.469121] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 231.600799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 247.555888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 247.816319] Lustre: lustre-OST0001: new disk, initializing [ 247.828462] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 247.838781] Lustre: 9542:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 247.996203] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 253.985648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 253.997789] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 254.090367] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 256.031459] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 270.090363] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 279.048453] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 288.036551] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing check_logdir /tmp/testlogs/ [ 293.780899] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing yml_node [ 298.593741] Lustre: DEBUG MARKER: Client: 2.17.57.108 [ 301.781992] Lustre: DEBUG MARKER: MDS: 2.17.57.108 [ 304.244685] Lustre: DEBUG MARKER: OSS: 2.17.57.108 [ 306.123392] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 21:36:18 EDT 2026 [ 325.626500] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 327.517151] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 346.080950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.092811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 346.113914] Lustre: Skipped 1 previous similar message [ 346.124590] Lustre: Skipped 3 previous similar messages [ 351.043344] Lustre: server umount lustre-MDT0000 complete [ 356.252215] LustreError: 6522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787881031 with bad export cookie 1224031996253307944 [ 356.256943] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 356.263405] LustreError: 6522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 356.320744] LustreError: 8476:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 356.323761] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.326086] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 356.349102] LustreError: 8476:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 356.365600] Lustre: Skipped 3 previous similar messages [ 361.452613] LustreError: 6538:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 361.455059] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 361.490886] LustreError: 6538:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 361.527659] Lustre: Skipped 2 previous similar messages [ 362.911802] Lustre: server umount lustre-MDT0001 complete [ 368.936249] Lustre: server umount lustre-OST0000 complete [ 374.406948] Lustre: server umount lustre-OST0001 complete [ 383.393599] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 395.113441] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 439.740905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 440.035391] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 440.091587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 440.209545] Lustre: lustre-MDT0000: new disk, initializing [ 440.359528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 440.381784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 444.874509] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 457.186604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 457.406469] Lustre: 15100:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 457.593854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 457.603678] Lustre: Skipped 1 previous similar message [ 457.759554] Lustre: lustre-MDT0001: new disk, initializing [ 457.975399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 458.136975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 458.157838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 464.268957] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 478.624974] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 487.042491] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 496.053910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 496.369437] Lustre: lustre-OST0000: new disk, initializing [ 496.376280] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 496.386580] Lustre: 17070:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 496.563421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 497.842399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 497.855137] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 498.117182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 504.928098] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 517.394680] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 523.442660] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 523.870677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 529.269565] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 529.518824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 533.990302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 534.003468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 534.012427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 534.033889] Lustre: Skipped 1 previous similar message [ 539.109214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 539.129128] Lustre: Skipped 1 previous similar message [ 546.271187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 546.433331] Lustre: server umount lustre-OST0000 complete [ 554.465569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.469406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 554.477532] Lustre: Skipped 2 previous similar messages [ 554.491626] Lustre: Skipped 3 previous similar messages [ 556.885343] Lustre: server umount lustre-MDT0000 complete [ 559.594079] LustreError: 15106:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 559.617771] LustreError: 15106:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 560.995162] LustreError: 18613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787881235 with bad export cookie 1224031996253309484 [ 561.000840] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 561.006771] LustreError: 18613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 561.427981] Lustre: server umount lustre-MDT0001 complete [ 567.510619] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 21:40:39 (1787881239) [ 585.227764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 585.698386] LustreError: 19946:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.772867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 589.961737] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 590.821953] LustreError: 19947:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.059482] LustreError: 19946:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.095593] LustreError: 19946:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 601.565211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 606.817907] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 616.706094] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 623.758649] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 626.104290] Lustre: Failing over lustre-MDT0000 [ 626.474477] Lustre: server umount lustre-MDT0000 complete [ 626.658924] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 626.687510] LustreError: 19943:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 626.714837] LustreError: 19943:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 644.063122] Lustre: 3665:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787881302/real 1787881302] req@ffff89fcdea70f00 x1874728864359808/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787881318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 644.078742] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 645.200935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 653.344842] LustreError: 3661:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff89fcdea730c0 x1874728864362496/t0(0) o250->MGC192.168.204.131@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 653.709685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 653.721455] Lustre: Skipped 1 previous similar message [ 653.791293] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 658.411292] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 658.918300] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 658.924972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 658.955643] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 668.729042] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 674.273560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 674.279135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 674.289249] Lustre: Skipped 1 previous similar message [ 674.299727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 679.668180] Lustre: server umount lustre-MDT0000 complete [ 682.477168] LustreError: 19941:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 682.511755] LustreError: 19941:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 17 previous similar messages [ 683.492639] LustreError: 19928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787881358 with bad export cookie 1224031996253310898 [ 683.497054] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 683.501425] LustreError: 19928:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 683.818031] Lustre: server umount lustre-MDT0001 complete [ 693.422979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.900971] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 707.473556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 713.606311] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 721.422423] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 726.743774] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 728.924249] Lustre: Failing over lustre-MDT0000 [ 729.457976] Lustre: server umount lustre-MDT0000 complete [ 729.570974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 729.584070] Lustre: Skipped 2 previous similar messages [ 745.887102] Lustre: 3662:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787881404/real 1787881404] req@ffff89fcdea72d00 x1874728864389504/t0(0) o400->MGC192.168.204.131@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787881420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 745.914169] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 748.316671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 751.072541] LustreError: 24016:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 751.086249] LustreError: 24016:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 14 previous similar messages [ 755.387639] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 759.484026] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 760.807932] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 760.822757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 760.828454] Lustre: Skipped 1 previous similar message [ 760.842881] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 769.075025] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 776.161024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 776.169035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 776.170903] Lustre: Skipped 1 previous similar message [ 776.180866] Lustre: Skipped 3 previous similar messages [ 779.661646] Lustre: server umount lustre-MDT0000 complete [ 783.532835] LustreError: 23268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787881458 with bad export cookie 1224031996253311836 [ 783.544208] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 783.556646] LustreError: 23268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 784.023213] Lustre: server umount lustre-MDT0001 complete [ 794.888297] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 21:44:26 (1787881466) [ 813.808160] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 824.093986] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 864.404853] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 876.802706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 877.115838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 877.166524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 877.310938] Lustre: lustre-MDT0000: new disk, initializing [ 877.420905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.428669] Lustre: Skipped 3 previous similar messages [ 877.453798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 882.004498] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 893.280353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.429702] Lustre: 29659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 893.740314] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 898.748206] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 908.353851] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 915.863222] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 924.150875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 924.478347] Lustre: 31632:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 926.206994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 926.271978] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 926.375772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 926.394427] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 933.391514] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 943.169532] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 950.004524] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 972.771931] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 972.794302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 972.802026] Lustre: Skipped 2 previous similar messages [ 977.378098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 987.103269] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 987.359784] Lustre: server umount lustre-OST0000 complete [ 998.777980] Lustre: server umount lustre-MDT0000 complete [ 1002.816429] LustreError: 29651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787881677 with bad export cookie 1224031996253312242 [ 1002.826174] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.833441] LustreError: 29651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1003.308123] Lustre: server umount lustre-MDT0001 complete [ 1015.679422] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1018.998217] Key type lgssc unregistered [ 1019.240599] LNet: 33617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.246789] LNetError: 33617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.266704] LNet: Removed LNI 192.168.204.131@tcp [ 1020.207130] Key type .llcrypt unregistered [ 1020.209756] Key type ._llcrypt unregistered [ 1051.790578] Key type ._llcrypt registered [ 1051.792975] Key type .llcrypt registered [ 1051.995372] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1053.390585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1053.751652] alg: No test for adler32 (adler32-zlib) [ 1054.932107] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 1055.262336] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1057.049240] Key type lgssc registered [ 1058.428562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1066.816225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1066.849853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.449723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1073.079566] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1078.754711] LustreError: 35033:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1078.779903] LustreError: 35033:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1082.768646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1083.174449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1087.783351] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1097.333308] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1104.517567] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1111.557598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1112.156594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1120.115155] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1127.554655] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1129.646654] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1145.899832] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 21:50:18 (1787881818) [ 1148.000700] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1150.306311] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1157.877946] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 21:50:29 (1787881829) [ 1168.478116] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 21:50:40 (1787881840) [ 1169.591873] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1179.141632] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1179.147823] Lustre: Skipped 2 previous similar messages [ 1181.341121] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 21:50:53 (1787881853) [ 1183.610431] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1183.617847] Lustre: Skipped 2 previous similar messages [ 1191.935791] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1191.950262] Lustre: Skipped 4 previous similar messages [ 1208.467291] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 21:51:20 (1787881880) [ 1211.639934] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1215.885388] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1225.721592] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 21:51:37 (1787881897) [ 1227.617205] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1227.629934] Lustre: Skipped 6 previous similar messages [ 1242.076575] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 21:51:54 (1787881914) [ 1261.041697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1261.048558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.056951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1261.079204] Lustre: Skipped 1 previous similar message [ 1264.798792] Lustre: server umount lustre-OST0000 complete [ 1271.264414] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1271.265802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1271.283746] Lustre: Skipped 1 previous similar message [ 1271.302049] Lustre: Skipped 2 previous similar messages [ 1276.055428] Lustre: server umount lustre-MDT0000 complete [ 1276.384599] LustreError: 35032:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.400498] LustreError: 35032:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1280.014647] LustreError: 35014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787881954 with bad export cookie 4163139136895714511 [ 1280.018461] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1280.035050] LustreError: 35014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1280.312778] Lustre: server umount lustre-MDT0001 complete [ 1292.816850] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1295.842900] Key type lgssc unregistered [ 1296.164781] LNet: 42895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1296.176986] LNetError: 42895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.205513] LNet: Removed LNI 192.168.204.131@tcp [ 1297.283650] Key type .llcrypt unregistered [ 1297.287520] Key type ._llcrypt unregistered [ 1315.651456] Key type ._llcrypt registered [ 1315.657565] Key type .llcrypt registered [ 1315.816702] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 1332.439716] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1333.638041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1333.698865] alg: No test for adler32 (adler32-zlib) [ 1334.729147] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 1334.971245] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1336.729086] Key type lgssc registered [ 1338.084733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1376.945772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1376.988574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1378.249499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1378.274738] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1378.345810] Lustre: lustre-MDT0000: new disk, initializing [ 1378.420766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1378.445148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1383.766839] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1395.226659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.328051] Lustre: 46047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1395.373952] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1395.377526] Lustre: Skipped 1 previous similar message [ 1395.435326] Lustre: lustre-MDT0001: new disk, initializing [ 1395.512030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1395.546765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1395.560297] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1400.209475] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1411.309920] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1418.527541] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1425.647902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1425.913988] Lustre: lustre-OST0000: new disk, initializing [ 1425.917663] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1425.924617] Lustre: 48016:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1426.015411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1432.124476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1432.143986] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1432.289817] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1432.731123] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1443.012771] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1447.763750] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1447.975282] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1453.090711] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1453.446087] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1458.160632] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1458.170329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1458.174506] LustreError: Skipped 1 previous similar message [ 1458.182936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1458.194663] Lustre: Skipped 1 previous similar message [ 1458.222898] Lustre: Skipped 1 previous similar message [ 1461.894439] Lustre: server umount lustre-OST0000 complete [ 1467.363549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1467.373567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.382362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1468.391014] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1468.391528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1471.721351] Lustre: server umount lustre-MDT0000 complete [ 1475.244635] LustreError: 46039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787882150 with bad export cookie 6257339708138924406 [ 1475.245687] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1475.259954] LustreError: 46039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1475.597303] Lustre: server umount lustre-MDT0001 complete [ 1484.465285] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 21:55:56 (1787882156) [ 1490.830758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1491.289316] LustreError: 50477:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1491.322716] LustreError: 50477:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1491.365947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1495.227290] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1496.544689] LustreError: 50478:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1501.685089] LustreError: 50477:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1503.766397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1504.209248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1509.661063] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1518.600482] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1525.129348] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1534.539251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1535.224082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1542.071742] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1550.170841] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1554.524865] LustreError: 53006:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1555.356489] Lustre: Modifying parameter general.timeout=1 in log params [ 1556.965595] Lustre: Modifying parameter general.timeout=3 in log params [ 1556.970219] Lustre: Skipped 1 previous similar message [ 1559.417656] Lustre: Modifying parameter general.timeout=6 in log params [ 1559.422211] Lustre: Skipped 2 previous similar messages [ 1563.955292] Lustre: Modifying parameter general.timeout=11 in log params [ 1563.958025] Lustre: Skipped 4 previous similar messages [ 1572.485063] Lustre: Modifying parameter general.timeout=20 in log params [ 1572.494808] Lustre: Skipped 8 previous similar messages [ 1588.739848] Lustre: Modifying parameter general.timeout=38 in log params [ 1588.744787] Lustre: Skipped 17 previous similar messages [ 1621.161562] Lustre: Modifying parameter general.timeout=76 in log params [ 1621.167126] Lustre: Skipped 37 previous similar messages [ 1680.984843] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 21:59:12 (1787882352) [ 1684.256432] Lustre: 59952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1685.330804] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1685.341819] Lustre: Skipped 56 previous similar messages [ 1686.261673] Lustre: 60050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1688.302408] Lustre: 60148:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule3 jobid={cat.0.0} rate=1000 (mode = 2) failed: rc = -17 [ 1694.124673] Lustre: 60463:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1694.130835] Lustre: 60463:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1722.856161] Lustre: 61826:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1722.884759] LustreError: 61826:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1728.206646] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 21:59:59 (1787882399) [ 1744.864640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1744.879619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1744.900991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1744.908987] Lustre: Skipped 1 previous similar message [ 1749.302263] Lustre: server umount lustre-MDT0000 complete [ 1752.034554] LustreError: 52114:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1752.065617] LustreError: 52114:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1753.067673] LustreError: 53055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787882427 with bad export cookie 6257339708138925288 [ 1753.073251] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1753.081802] LustreError: 53055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1753.418637] Lustre: server umount lustre-MDT0001 complete [ 1767.808761] Lustre: server umount lustre-OST0000 complete [ 1796.696827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.053396] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1797.294656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.002724] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1811.613406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1811.711618] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1811.733444] Lustre: 64374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1817.700480] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1826.724630] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1833.668585] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1842.233934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1842.526028] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1842.765742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1842.769822] Lustre: Skipped 1 previous similar message [ 1849.336942] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1857.258596] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1858.996593] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1859.005791] Lustre: Skipped 20 previous similar messages [ 1859.044430] Lustre: 66941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1866.728831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1866.748434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1866.760964] Lustre: Skipped 2 previous similar messages [ 1866.770139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1866.777622] Lustre: Skipped 3 previous similar messages [ 1871.883391] Lustre: server umount lustre-OST0000 complete [ 1877.472751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.477256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1877.504103] Lustre: Skipped 2 previous similar messages [ 1877.515344] Lustre: Skipped 3 previous similar messages [ 1881.502508] Lustre: server umount lustre-MDT0000 complete [ 1882.594240] LustreError: 65124:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1882.604758] LustreError: 65124:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1884.420118] LustreError: 65126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787882559 with bad export cookie 6257339708138928347 [ 1884.426085] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1884.428747] LustreError: 65126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1884.666252] Lustre: server umount lustre-MDT0001 complete [ 1895.352795] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 1898.141494] Key type lgssc unregistered [ 1898.413410] LNet: 68155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1898.426542] LNetError: 68155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1898.444575] LNet: Removed LNI 192.168.204.131@tcp [ 1899.188245] Key type .llcrypt unregistered [ 1899.191761] Key type ._llcrypt unregistered [ 1922.780853] Key type ._llcrypt registered [ 1922.783558] Key type .llcrypt registered [ 1922.890553] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 1924.076324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1924.145564] alg: No test for adler32 (adler32-zlib) [ 1925.271366] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 1925.550927] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 1927.223680] Key type lgssc registered [ 1928.411752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1935.799725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1935.820158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1937.241638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.194365] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1947.622277] LustreError: 69571:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1947.648770] LustreError: 69571:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1949.152357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1949.678287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1953.642546] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1961.828733] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1968.567613] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1976.034570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1976.607763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.285846] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 1990.752461] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2001.296545] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2010.599651] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.604537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2010.609122] Lustre: Skipped 2 previous similar messages [ 2010.624568] Lustre: Skipped 2 previous similar messages [ 2015.714791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2015.735890] Lustre: Skipped 2 previous similar messages [ 2016.234799] Lustre: server umount lustre-MDT0000 complete [ 2020.485905] LustreError: 69552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787882695 with bad export cookie 4405268799499103659 [ 2020.495404] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2020.501176] LustreError: 69552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2020.939749] Lustre: server umount lustre-MDT0001 complete [ 2035.791062] Lustre: server umount lustre-OST0000 complete [ 2062.152536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2062.465507] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2062.500542] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2062.737208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2067.727645] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2076.900578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2077.029659] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2077.058959] Lustre: 74440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2077.574486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2083.210233] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2090.959380] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2098.152616] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2105.502681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2105.654175] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2106.239042] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2113.518241] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2122.669362] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2125.024903] Lustre: 77016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2127.559008] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2138.086804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2138.087413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.102284] LustreError: Skipped 1 previous similar message [ 2138.105974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2138.114905] Lustre: Skipped 1 previous similar message [ 2138.125900] Lustre: Skipped 1 previous similar message [ 2139.689934] Lustre: server umount lustre-OST0000 complete [ 2148.325696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2148.351701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2148.366737] Lustre: Skipped 1 previous similar message [ 2150.666679] Lustre: server umount lustre-MDT0000 complete [ 2153.440805] LustreError: 74452:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2153.467780] LustreError: 74452:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2155.057653] LustreError: 75187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787882829 with bad export cookie 4405268799499104492 [ 2155.065201] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2155.075279] LustreError: 75187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2155.385839] Lustre: server umount lustre-MDT0001 complete [ 2167.347572] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2170.223553] Key type lgssc unregistered [ 2170.533546] LNet: 78230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.544481] LNetError: 78230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2171.623138] LNet: Removed LNI 192.168.204.131@tcp [ 2172.590409] Key type .llcrypt unregistered [ 2172.594078] Key type ._llcrypt unregistered [ 2199.914405] Key type ._llcrypt registered [ 2199.920432] Key type .llcrypt registered [ 2200.105141] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2201.840440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2201.876800] alg: No test for adler32 (adler32-zlib) [ 2203.076519] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 2203.324323] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2204.983562] Key type lgssc registered [ 2206.194818] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2215.328313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2215.360213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.032582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.017831] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2226.158789] LustreError: 79645:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2226.196202] LustreError: 79645:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2231.276629] LustreError: 79646:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2231.433332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2232.549257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2238.975858] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2248.533997] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2255.397424] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2263.593971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2264.523869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2271.359724] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2279.307905] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2295.341871] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2296.672551] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2297.991428] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2300.390781] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2300.394593] Lustre: Skipped 1 previous similar message [ 2304.695435] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2304.702850] Lustre: Skipped 4 previous similar messages [ 2312.835398] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2312.840856] Lustre: Skipped 5 previous similar messages [ 2329.394447] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2329.399128] Lustre: Skipped 15 previous similar messages [ 2361.737267] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2361.744760] Lustre: Skipped 35 previous similar messages [ 2400.355293] Lustre: 87779:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2403.616961] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 22:11:15 (1787883075) [ 2405.212291] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2407.187757] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 22:11:19 (1787883079) [ 2415.072331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2415.076724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2415.084165] LustreError: Skipped 1 previous similar message [ 2415.093297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2415.098300] Lustre: Skipped 1 previous similar message [ 2415.110281] Lustre: Skipped 1 previous similar message [ 2420.194666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2425.311137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2425.316881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2425.343828] Lustre: Skipped 2 previous similar messages [ 2425.551485] Lustre: server umount lustre-OST0000 complete [ 2430.441799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.475949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2430.485031] Lustre: Skipped 1 previous similar message [ 2435.553439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2435.561925] Lustre: Skipped 1 previous similar message [ 2444.255236] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2444.469478] Lustre: server umount lustre-MDT0000 complete [ 2445.793763] LustreError: 79645:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2445.817538] LustreError: 79645:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2448.811804] LustreError: 79626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787883123 with bad export cookie 11549693760589269623 [ 2448.814065] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2448.826761] LustreError: 79626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2449.281356] Lustre: server umount lustre-MDT0001 complete [ 2462.676465] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2465.817529] Key type lgssc unregistered [ 2466.185906] LNet: 89033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2466.204154] LNetError: 89033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2466.225324] LNet: Removed LNI 192.168.204.131@tcp [ 2467.378848] Key type .llcrypt unregistered [ 2467.380760] Key type ._llcrypt unregistered [ 2473.775130] Key type ._llcrypt registered [ 2473.776847] Key type .llcrypt registered [ 2473.975991] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2480.543975] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules [ 2482.508062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2482.649201] alg: No test for adler32 (adler32-zlib) [ 2484.258510] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 2484.687578] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2486.463894] Key type lgssc registered [ 2488.376310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2490.596942] LustreError: 90309:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2509.107512] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2511.596761] LustreError: 90309:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2519.002735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2519.039628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2520.725023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2525.360593] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2530.273198] LustreError: 91306:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.298396] LustreError: 91306:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2535.405419] LustreError: 91305:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2535.659260] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 22:13:27 (1787883207) [ 2540.523367] LustreError: 91306:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.958536] LustreError: 91337:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2540.982521] LustreError: 91337:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2540.996744] LustreError: 91337:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 2541.535718] Lustre: server umount lustre-MDT0000 complete [ 2557.133474] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 2560.453625] Key type lgssc unregistered [ 2560.734584] LNet: 92835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2560.742599] LNetError: 92835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2560.759804] LNet: Removed LNI 192.168.204.131@tcp [ 2561.735912] Key type .llcrypt unregistered [ 2561.738858] Key type ._llcrypt unregistered [ 2580.702150] Key type ._llcrypt registered [ 2580.704035] Key type .llcrypt registered [ 2580.815893] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2581.727454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2581.740565] alg: No test for adler32 (adler32-zlib) [ 2582.786872] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 2583.103122] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 2584.799236] Key type lgssc registered [ 2585.830399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2593.460672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2593.490161] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2595.000184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2599.623186] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2604.522513] LustreError: 94053:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.556706] LustreError: 94053:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2609.438993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.641469] LustreError: 94052:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2610.583261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2615.907482] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2625.766217] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2631.230711] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2638.913709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2639.693100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2647.460785] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2656.565364] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2711.969867] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 22:16:23 (1787883383) [ 2714.242611] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2717.146281] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 22:16:28 (1787883388) [ 2722.785446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2722.804484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.823445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2725.868691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2725.868768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2725.889208] Lustre: Skipped 1 previous similar message [ 2725.903072] Lustre: Skipped 2 previous similar messages [ 2728.282078] Lustre: server umount lustre-MDT0000 complete [ 2730.978079] LustreError: 94047:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2731.007668] LustreError: 94047:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2732.191602] LustreError: 95907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787883407 with bad export cookie 17551487899971417498 [ 2732.195272] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2732.208072] LustreError: 95907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2732.504409] Lustre: server umount lustre-MDT0001 complete [ 2747.296169] Lustre: server umount lustre-OST0000 complete [ 2760.092517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2760.587312] LustreError: 98366:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2760.724160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2765.433485] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2765.797026] LustreError: 98367:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2775.262794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2776.129084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2780.802229] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2791.567837] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2799.466261] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2813.857771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2814.060499] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2814.072194] LustreError: 98355:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2814.098965] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2814.107634] LustreError: 100450:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2814.113943] LustreError: 100450:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2814.123821] LustreError: 100450:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2814.205858] Lustre: server umount lustre-OST0000 complete [ 2814.209660] LustreError: 100450:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2822.976510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2823.201973] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2823.210029] LustreError: 98356:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2823.238408] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2823.249890] LustreError: 100913:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2823.255902] LustreError: 100913:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2823.261824] LustreError: 100913:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2823.331571] Lustre: server umount lustre-OST0000 complete [ 2823.334814] LustreError: 100913:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2832.450924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2832.667916] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2832.674387] Lustre: Found index 0 for lustre-OST0000, updating log [ 2832.680745] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2832.750676] Lustre: lustre-OST0000: new disk, initializing [ 2832.756355] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2832.770830] Lustre: 101447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2833.093486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2840.510730] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2844.819888] Lustre: Failing over lustre-OST0000 [ 2844.993995] Lustre: server umount lustre-OST0000 complete [ 2849.014079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2849.037955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2854.371462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2854.377696] Lustre: Skipped 2 previous similar messages [ 2855.377647] Lustre: server umount lustre-MDT0000 complete [ 2859.490550] LustreError: 98361:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2859.514654] LustreError: 98361:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2859.593017] LustreError: 98346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787883534 with bad export cookie 17551487899971420242 [ 2859.598677] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2859.611326] LustreError: 98346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2859.997144] Lustre: server umount lustre-MDT0001 complete [ 2871.211826] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 22:19:02 (1787883542) [ 2893.864251] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 2906.066185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2906.567770] LustreError: 103817:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2906.735671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2912.113079] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2922.328886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2928.471944] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2932.092591] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2939.082632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2940.002840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.019260] Lustre: Skipped 1 previous similar message [ 2947.371473] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2956.256371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2956.378913] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2956.463702] Lustre: lustre-OST0001: new disk, initializing [ 2956.467655] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2956.476438] Lustre: 106139:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2962.523309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2962.546494] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2962.653839] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2963.032959] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 2970.464478] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2980.082700] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2986.175093] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [ 2988.516076] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2988.535710] Lustre: Skipped 3 previous similar messages [ 2991.785956] Lustre: server umount lustre-MDT0001 complete [ 2993.648149] LustreError: 103817:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2993.670420] LustreError: 103817:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 3002.377379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3002.472549] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3002.479242] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3002.531795] Lustre: 103806:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3004.270410] LustreError: 107718:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3004.288794] LustreError: 107718:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3004.300799] LustreError: 103809:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3004.369899] LustreError: 107723:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.131@tcp: cfg command failed: rc = -17 [ 3004.375805] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3004.375805] [ 3007.735172] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3008.502239] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 0@lo (at 0@lo) [ 3018.905496] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 22:21:30 (1787883690) [ 3036.744678] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3057.915737] Lustre: 109753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3090.498513] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3097.695629] Lustre: 110890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3202.528979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3202.547493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3202.583498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3202.595598] Lustre: Skipped 4 previous similar messages [ 3204.727807] Lustre: server umount lustre-MDT0000 complete [ 3208.172942] LustreError: 105244:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3208.179905] LustreError: 105244:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [ 3209.877062] LustreError: 103798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787883884 with bad export cookie 17551487899971421166 [ 3209.882086] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3209.908197] LustreError: 103798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3210.464371] Lustre: server umount lustre-MDT0001 complete [ 3228.158884] Lustre: server umount lustre-OST0000 complete [ 3229.152431] Lustre: 93364:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787883888/real 1787883888] req@ffff89fcebd2a580 x1874731474465792/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787883904 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3229.179448] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3229.192447] Lustre: Skipped 3 previous similar messages [ 3230.497096] Lustre: 93366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787883889/real 1787883889] req@ffff89fcffbc6580 x1874731474466048/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787883905 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3232.946502] Lustre: server umount lustre-OST0001 complete [ 3239.651786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3249.854034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3259.698872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3271.411258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3297.702655] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3308.961323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.978575] Lustre: lustre-MDT0000: reset Object Index mappings [ 3309.689091] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3309.703226] Lustre: Skipped 2 previous similar messages [ 3314.912875] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3326.196224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.217372] Lustre: lustre-MDT0001: reset Object Index mappings [ 3333.616574] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3337.261444] Lustre: 116013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3345.780490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3352.113886] LustreError: 116368:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3352.131308] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:200 to 0x280000400:225) [ 3352.143436] LustreError: 116368:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 3353.854445] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3354.171288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:349 to 0x280000401:386) [ 3363.474440] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3369.465309] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:200 to 0x2c0000400:225) [ 3369.478469] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:156 to 0x2c0000401:193) [ 3370.282500] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3377.536745] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3381.286410] Lustre: 117890:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3432.894931] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 22:28:24 (1787884104) [ 3459.554734] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.565704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3459.575149] Lustre: Skipped 1 previous similar message [ 3459.576466] Lustre: Skipped 4 previous similar messages [ 3465.048634] Lustre: server umount lustre-MDT0000 complete [ 3468.836751] LustreError: 114843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787884143 with bad export cookie 17551487899971468094 [ 3468.860551] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3475.325073] Lustre: server umount lustre-MDT0001 complete [ 3479.097214] LustreError: 118648:0:(ldlm_resource.c:1207:ldlm_resource_complain()) filter-lustre-OST0000_UUID: namespace resource [0x280000401:0xf6:0x0].0x0 (ffff89fcc523b400) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3493.343142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3493.492075] Lustre: server umount lustre-OST0000 complete [ 3503.951053] Lustre: server umount lustre-OST0001 complete [ 3513.371215] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 3529.949734] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3569.769127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3570.273436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3570.353473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3570.534907] Lustre: lustre-MDT0000: new disk, initializing [ 3570.636319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3570.642642] Lustre: Skipped 3 previous similar messages [ 3570.719056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3575.560092] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3582.937505] Lustre: server umount lustre-MDT0000 complete [ 3592.790550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.141815] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3593.160192] Lustre: 122182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3593.174319] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3599.092397] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3604.350605] Lustre: server umount lustre-MDT0000 complete [ 3613.433811] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:31:25 (1787884285) [ 3615.593544] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3618.227770] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:31:29 (1787884289) [ 3638.325260] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 3651.338727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3658.045498] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3669.059597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3669.250485] Lustre: 124235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3669.337579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3669.341646] Lustre: Skipped 1 previous similar message [ 3669.445444] Lustre: lustre-MDT0001: new disk, initializing [ 3669.661941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3669.682561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3676.094911] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3682.139823] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3690.136069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3690.487450] Lustre: lustre-OST0000: new disk, initializing [ 3690.494459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3690.499351] Lustre: 125816:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3692.294916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3692.304791] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3692.523662] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3698.647923] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3709.781375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3710.065591] Lustre: lustre-OST0001: new disk, initializing [ 3710.074289] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3710.087580] Lustre: 126686:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3711.186479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3711.195066] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3711.333316] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3716.864280] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 3727.603824] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3731.832679] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4148.759550] Lustre: 133104:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4148.765460] Lustre: 133104:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4148.770989] Lustre: 133104:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4148.779631] Lustre: 133104:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4148.789425] Lustre: 133104:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4148.796233] Lustre: 133104:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4441.158366] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:45:11 (1787885111) [ 4457.952094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4457.961302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.977021] Lustre: Skipped 4 previous similar messages [ 4457.984585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4457.996056] Lustre: Skipped 11 previous similar messages [ 4462.967533] Lustre: server umount lustre-MDT0000 complete [ 4464.113148] LustreError: 127502:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4464.142868] LustreError: 127502:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 4467.412270] LustreError: 124229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787885142 with bad export cookie 17551487899971536120 [ 4467.415405] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4467.421461] LustreError: 124229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4469.226827] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4469.247720] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4469.251576] Lustre: Skipped 4 previous similar messages [ 4469.270915] Lustre: Skipped 5 previous similar messages [ 4474.116530] Lustre: server umount lustre-MDT0001 complete [ 4484.724951] Lustre: server umount lustre-OST0000 complete [ 4495.078741] Lustre: server umount lustre-OST0001 complete [ 4502.951406] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 4514.324643] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 4558.138118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4558.364652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4558.409598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4558.469845] Lustre: lustre-MDT0000: new disk, initializing [ 4558.532271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4558.536724] Lustre: Skipped 5 previous similar messages [ 4558.548383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4563.917945] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4575.966301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4576.087714] Lustre: 136170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4576.156607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4576.167533] Lustre: Skipped 1 previous similar message [ 4576.257701] Lustre: lustre-MDT0001: new disk, initializing [ 4576.420515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4576.440391] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4581.523360] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4592.862587] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4598.999520] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4607.566690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4607.884868] Lustre: lustre-OST0000: new disk, initializing [ 4607.889845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4607.897513] Lustre: 138136:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4607.989516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4607.995507] Lustre: Skipped 1 previous similar message [ 4609.812410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4609.836621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4609.961236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4614.731036] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 4625.104249] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4635.834597] Lustre: lustre-MDD0000: changelog on [ 4639.359261] Lustre: lustre-MDD0001: changelog on [ 6749.562908] Lustre: lustre-MDD0001: changelog off [ 6751.673974] Lustre: 141683:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6752.552875] Lustre: lustre-MDD0000: changelog off [ 6760.416910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6760.419754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6760.433440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6760.433447] Lustre: Skipped 2 previous similar messages [ 6760.472236] Lustre: Skipped 1 previous similar message [ 6764.790704] Lustre: server umount lustre-OST0000 complete [ 6770.659111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6770.660946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6770.672544] Lustre: Skipped 1 previous similar message [ 6770.681177] Lustre: Skipped 1 previous similar message [ 6775.275636] Lustre: server umount lustre-MDT0000 complete [ 6775.777567] LustreError: 139594:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6775.790805] LustreError: 139594:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 6778.840761] LustreError: 138148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787887453 with bad export cookie 17551487899972310509 [ 6778.849173] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6778.857193] LustreError: 138148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6779.192090] Lustre: server umount lustre-MDT0001 complete [ 6790.775981] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 6793.578655] Key type lgssc unregistered [ 6793.858608] LNet: 147332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6793.870610] LNetError: 147332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6794.918202] LNet: Removed LNI 192.168.204.131@tcp [ 6795.911183] Key type .llcrypt unregistered [ 6795.913215] Key type ._llcrypt unregistered [ 6816.949916] Key type ._llcrypt registered [ 6816.959348] Key type .llcrypt registered [ 6817.103585] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 6833.473344] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 6834.681509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6834.812722] alg: No test for adler32 (adler32-zlib) [ 6835.913988] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 6836.167453] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 6837.879236] Key type lgssc registered [ 6839.015310] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6876.097572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6876.123687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6877.539480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6877.627928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6877.746602] Lustre: lustre-MDT0000: new disk, initializing [ 6877.827613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6877.842316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6883.073581] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6896.638859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6896.891716] Lustre: 150533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6897.000719] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6897.016600] Lustre: Skipped 1 previous similar message [ 6897.280257] Lustre: lustre-MDT0001: new disk, initializing [ 6897.631377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6897.792058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6897.817751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6905.280268] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6916.328681] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6922.725923] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6930.623225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6930.868134] Lustre: lustre-OST0000: new disk, initializing [ 6930.873573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6930.881135] Lustre: 152501:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6930.994514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6940.241242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6940.247747] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6940.282314] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 6940.443712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6953.172527] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6960.981350] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_module obdecho/obdecho [ 6963.207284] LustreError: 153506:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6963.230727] LustreError: 153506:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6965.513787] LustreError: 153604:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6984.679076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6984.684795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6984.697677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6984.704515] Lustre: Skipped 2 previous similar messages [ 6984.734411] Lustre: Skipped 2 previous similar messages [ 6986.221342] Lustre: server umount lustre-MDT0000 complete [ 6990.601428] LustreError: 150526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787887665 with bad export cookie 8246032565478075501 [ 6990.614611] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6990.631721] LustreError: 150526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6990.968984] Lustre: server umount lustre-MDT0001 complete [ 7004.496520] Lustre: server umount lustre-OST0000 complete [ 7029.655349] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 7043.602289] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 7082.589321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7082.882994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7082.907316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7082.989750] Lustre: lustre-MDT0000: new disk, initializing [ 7083.084938] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7083.104754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7088.192508] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7099.458987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7099.573742] Lustre: 157323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7099.636052] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7099.640180] Lustre: Skipped 1 previous similar message [ 7099.713784] Lustre: lustre-MDT0001: new disk, initializing [ 7099.856273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7099.904678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7099.921529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7104.875250] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7114.807427] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7122.080830] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7130.469951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7130.805411] Lustre: lustre-OST0000: new disk, initializing [ 7130.817481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7130.839788] Lustre: 159295:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7130.953342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7139.745429] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7140.438555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7140.451728] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7140.664782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7150.178583] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7154.886588] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7155.055300] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7160.127347] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7160.340900] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7166.439259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7166.441922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7166.450831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7166.453616] LustreError: Skipped 1 previous similar message [ 7166.492571] Lustre: Skipped 1 previous similar message [ 7168.550928] Lustre: server umount lustre-OST0000 complete [ 7176.672452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7176.679390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7176.683554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7176.694315] Lustre: Skipped 1 previous similar message [ 7176.725873] Lustre: Skipped 2 previous similar messages [ 7178.704028] Lustre: server umount lustre-MDT0000 complete [ 7181.804518] LustreError: 157330:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7181.825587] LustreError: 157330:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 7182.616571] LustreError: 157315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787887857 with bad export cookie 8246032565478076460 [ 7182.618106] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7182.630470] LustreError: 157315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7182.986424] Lustre: server umount lustre-MDT0001 complete [ 7186.705862] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 23:30:58 (1787887858) [ 7188.427806] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7190.129114] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 23:31:02 (1787887862) [ 7198.258214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7198.674365] LustreError: 161706:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7198.742300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7203.391138] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7203.810392] LustreError: 161707:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7208.929428] LustreError: 161706:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7214.052093] LustreError: 161707:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7214.469746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7221.584055] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7228.950900] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7234.524747] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7242.107427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7242.606448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7242.611929] Lustre: Skipped 1 previous similar message [ 7248.899832] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7256.865723] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7270.373139] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7270.375150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7270.400837] Lustre: Skipped 1 previous similar message [ 7270.415119] Lustre: Skipped 2 previous similar messages [ 7275.498928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7275.513018] Lustre: Skipped 3 previous similar messages [ 7275.780900] Lustre: server umount lustre-MDT0000 complete [ 7278.205249] LustreError: 161701:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7281.498975] LustreError: 161686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787887956 with bad export cookie 8246032565478077342 [ 7281.514674] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7285.729238] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7285.750779] Lustre: Skipped 1 previous similar message [ 7285.757362] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7285.767498] Lustre: Skipped 1 previous similar message [ 7288.230465] Lustre: server umount lustre-MDT0001 complete [ 7295.745801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7310.303374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7310.548725] LustreError: 165570:0:(llog.c:1655:llog_backup()) MGC192.168.204.131@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7310.556401] Lustre: 165570:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.131@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7311.329448] LustreError: 165575:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.351608] LustreError: 165575:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 7343.737357] LustreError: 165576:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7343.767417] LustreError: 165576:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 22 previous similar messages [ 7371.231978] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7371.244961] Lustre: 165570:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.131@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7409.272258] LustreError: 165577:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7409.296671] LustreError: 165577:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 42 previous similar messages [ 7412.215063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7417.323806] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7417.885476] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7428.961167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7436.770795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 7457.783218] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7458.673076] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7466.736249] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7468.816618] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7474.865927] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7476.307724] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7498.466886] Lustre: server umount lustre-MDT0000 complete [ 7498.725147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7498.727969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7501.783282] LustreError: 161688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787888176 with bad export cookie 8246032565478083831 [ 7501.785820] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7501.795275] LustreError: 161688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7502.188568] Lustre: server umount lustre-MDT0001 complete [ 7516.357402] Lustre: server umount lustre-OST0000 complete [ 7539.573224] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 7550.971115] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 7592.748802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7593.049614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7593.110577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7593.287143] Lustre: lustre-MDT0000: new disk, initializing [ 7593.405402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7593.412312] Lustre: Skipped 1 previous similar message [ 7593.431772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7597.634241] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7609.797243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7609.884810] Lustre: 170678:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7610.133952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7615.134622] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7626.014408] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7632.355518] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7639.944395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7640.208959] Lustre: lustre-OST0000: new disk, initializing [ 7640.212140] Lustre: Skipped 1 previous similar message [ 7640.219252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7640.225168] Lustre: Skipped 2 previous similar messages [ 7640.234817] Lustre: 172650:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7646.672990] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7649.849210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7649.855127] Lustre: Skipped 1 previous similar message [ 7649.862788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7649.992312] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7656.013798] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7661.182865] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7661.393730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7665.767919] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7666.045455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7670.764711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7670.769978] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7670.785225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7670.804722] Lustre: Skipped 3 previous similar messages [ 7670.835580] Lustre: Skipped 2 previous similar messages [ 7674.173399] Lustre: server umount lustre-OST0000 complete [ 7692.260175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7692.502903] Lustre: server umount lustre-MDT0000 complete [ 7695.330260] LustreError: 173545:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7695.360207] LustreError: 173545:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 41 previous similar messages [ 7696.467270] LustreError: 170672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787888371 with bad export cookie 8246032565478100631 [ 7696.474218] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7696.483757] LustreError: 170672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7697.238780] Lustre: server umount lustre-MDT0001 complete [ 7707.094207] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:39:38 (1787888378) [ 7713.676937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7718.432442] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7726.924265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7731.528931] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7740.082516] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7746.111756] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7752.799806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7758.605240] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7767.475743] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7778.369240] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:40:49 (1787888449) [ 7783.906489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7783.909817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7783.930048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7783.946196] Lustre: Skipped 3 previous similar messages [ 7783.958157] Lustre: Skipped 7 previous similar messages [ 7789.240322] Lustre: server umount lustre-OST0000 complete [ 7799.667799] Lustre: server umount lustre-MDT0000 complete [ 7803.730235] LustreError: 175042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787888478 with bad export cookie 8246032565478101506 [ 7803.745283] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7803.750445] LustreError: 175042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7804.282347] Lustre: server umount lustre-MDT0001 complete [ 7814.716915] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 7818.009357] Key type lgssc unregistered [ 7818.471772] LNet: 178795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7818.511804] LNetError: 178795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7818.554663] LNet: Removed LNI 192.168.204.131@tcp [ 7819.935744] Key type .llcrypt unregistered [ 7819.938593] Key type ._llcrypt unregistered [ 7842.744786] Key type ._llcrypt registered [ 7842.747758] Key type .llcrypt registered [ 7842.860276] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 7844.046935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7844.134038] alg: No test for adler32 (adler32-zlib) [ 7845.150332] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 7845.415107] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 7847.079229] Key type lgssc registered [ 7848.112764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7856.725881] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7856.746991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7872.290299] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7877.407520] LustreError: 180038:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.131@tcp: failed processing log, type 4: rc = -110 [ 7882.527513] LustreError: 180038:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7882.543256] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7882.550630] LustreError: MGC192.168.204.131@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7882.574417] LustreError: 180038:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7882.588153] LustreError: 180038:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7882.604193] LustreError: 180038:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7882.698595] Lustre: server umount lustre-OST0000 complete [ 7882.702539] LustreError: 180038:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7892.233851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7893.879937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7898.695862] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7903.211156] LustreError: 180537:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7903.251189] LustreError: 180537:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7908.326449] LustreError: 180538:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7909.806860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7910.274626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7915.514769] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7924.383349] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7932.304682] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7940.091040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7940.295507] LustreError: 182354:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7940.302769] Lustre: 182354:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.204.131@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7940.452028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7948.161976] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 7956.784842] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7961.059200] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7961.062873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7961.093733] Lustre: Skipped 1 previous similar message [ 7961.112785] Lustre: Skipped 2 previous similar messages [ 7965.926202] Lustre: server umount lustre-MDT0000 complete [ 7966.176991] LustreError: 180538:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7966.202994] LustreError: 180538:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7969.759177] LustreError: 180519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787888644 with bad export cookie 407026081833835856 [ 7969.766912] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7970.082828] Lustre: server umount lustre-MDT0001 complete [ 7983.534947] Lustre: server umount lustre-OST0000 complete [ 7996.073983] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 8008.162533] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8051.443281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8051.818753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8051.894356] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8052.023281] Lustre: lustre-MDT0000: new disk, initializing [ 8052.111685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8052.169183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8057.310626] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8069.666292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8069.821945] Lustre: 185882:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8069.906423] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8069.911963] Lustre: Skipped 1 previous similar message [ 8069.931128] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8070.023269] Lustre: lustre-MDT0001: new disk, initializing [ 8070.131439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8070.174903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8070.195440] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8075.620109] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8086.708972] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8093.545332] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8101.797563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8102.164921] Lustre: lustre-OST0000: new disk, initializing [ 8102.171795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8102.181888] Lustre: 187849:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8102.323379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8109.656424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8109.678648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8109.806561] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8111.001807] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8122.278576] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8127.386664] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8127.633671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8132.817620] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8133.095909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8135.652544] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8135.657077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8135.661031] Lustre: Skipped 2 previous similar messages [ 8135.662400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8135.692698] Lustre: Skipped 1 previous similar message [ 8140.771943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8140.781745] Lustre: Skipped 1 previous similar message [ 8142.032470] Lustre: server umount lustre-OST0000 complete [ 8146.912347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8146.916783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8146.944175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8151.010474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8152.305825] Lustre: server umount lustre-MDT0000 complete [ 8156.141382] LustreError: 188146:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8156.174198] LustreError: 188146:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8156.294843] LustreError: 185873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787888831 with bad export cookie 407026081833836570 [ 8156.296144] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8156.304610] LustreError: 185873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8156.593104] Lustre: server umount lustre-MDT0001 complete [ 8165.144451] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:47:17 (1787888837) [ 8182.934411] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 8186.030459] Key type lgssc unregistered [ 8186.398750] LNet: 190799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8186.406445] LNetError: 190799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8186.421557] LNet: Removed LNI 192.168.204.131@tcp [ 8187.319203] Key type .llcrypt unregistered [ 8187.322305] Key type ._llcrypt unregistered [ 8206.733819] Key type ._llcrypt registered [ 8206.736333] Key type .llcrypt registered [ 8206.844380] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8208.256506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8208.265845] alg: No test for adler32 (adler32-zlib) [ 8209.343385] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 8209.669959] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 8211.383530] Key type lgssc registered [ 8212.787094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8221.882447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8221.912519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8223.607898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8229.742740] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8233.968985] LustreError: 192018:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8233.992156] LustreError: 192018:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8239.076239] LustreError: 192019:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8241.098675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8241.585708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8248.739949] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8257.639152] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8263.966148] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8271.748674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8271.962891] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8271.965869] LustreError: 193883:0:(llog.c:1696:llog_backup()) MGC192.168.204.131@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8271.978097] Lustre: 193883:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.204.131@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8272.086568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8278.280932] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8285.613472] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8287.709632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8287.718681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8287.727739] Lustre: Skipped 2 previous similar messages [ 8287.739316] Lustre: Skipped 2 previous similar messages [ 8292.835190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8297.955612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8297.958912] Lustre: Skipped 3 previous similar messages [ 8302.047386] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8302.223477] Lustre: server umount lustre-MDT0000 complete [ 8303.076221] LustreError: 192018:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8303.101520] LustreError: 192018:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 8306.590042] LustreError: 192000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787888981 with bad export cookie 12717151302503568306 [ 8306.601064] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8306.606438] LustreError: 192000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8306.952157] Lustre: server umount lustre-MDT0001 complete [ 8321.826733] Lustre: server umount lustre-OST0000 complete [ 8332.367889] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 8343.630976] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8388.870342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8389.167877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8389.237667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8389.397940] Lustre: lustre-MDT0000: new disk, initializing [ 8389.522419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8389.538189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8395.044442] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8408.991382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8409.145925] Lustre: 197414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8409.216822] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8409.218341] Lustre: Skipped 1 previous similar message [ 8409.222756] LustreError: 198197:0:(llog.c:1696:llog_backup()) MGC192.168.204.131@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8409.227396] LustreError: 198197:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8409.231439] Lustre: 198197:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.131@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8409.249861] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8409.253903] Lustre: Skipped 1 previous similar message [ 8409.367828] Lustre: lustre-MDT0001: new disk, initializing [ 8409.551255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8409.612102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8409.634149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8414.970233] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8424.855746] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8429.600483] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8437.187465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8437.562861] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8437.565799] Lustre: Skipped 1 previous similar message [ 8437.568605] LustreError: 199359:0:(llog.c:1696:llog_backup()) MGC192.168.204.131@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8437.576908] LustreError: 199359:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 8437.583342] Lustre: 199359:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.204.131@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8437.593131] Lustre: 199359:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8437.614807] Lustre: lustre-OST0000: new disk, initializing [ 8437.618756] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8437.623935] Lustre: 199382:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8437.744335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8444.486256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8444.510203] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8444.649191] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8445.755587] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8457.391943] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8464.027444] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8464.411765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8470.449903] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8470.721237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8475.619128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8475.621780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8475.634069] Lustre: Skipped 1 previous similar message [ 8479.114443] Lustre: server umount lustre-OST0000 complete [ 8485.859171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8485.864757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8485.875597] Lustre: Skipped 1 previous similar message [ 8485.899090] Lustre: Skipped 2 previous similar messages [ 8489.791750] Lustre: server umount lustre-MDT0000 complete [ 8490.982625] LustreError: 197424:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8490.994700] LustreError: 197424:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8493.961114] LustreError: 197405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787889168 with bad export cookie 12717151302503569020 [ 8493.965409] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8493.973429] LustreError: 197405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8494.393738] Lustre: server umount lustre-MDT0001 complete [ 8504.258726] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:52:55 (1787889175) [ 8524.195647] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8534.463839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8534.813670] LustreError: 202510:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8534.862262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8539.291667] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8544.233233] LustreError: 202511:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8544.244425] LustreError: 202511:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8548.383301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8553.207863] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8556.151541] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8562.835651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8570.303785] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8579.897808] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8580.133639] Lustre: lustre-OST0001: new disk, initializing [ 8580.137645] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8580.148838] Lustre: 204818:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8580.271715] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8580.287449] Lustre: Skipped 2 previous similar messages [ 8585.908588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8585.921790] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8586.110497] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8588.453807] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8600.068475] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8609.033246] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8611.723092] Lustre: DEBUG MARKER: ADD OST3 [ 8617.627870] loop: module loaded [ 8617.700165] loop0: detected capacity change from 0 to 204800000 [ 8617.893732] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8617.912610] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8617.939209] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8625.629497] loop0: detected capacity change from 0 to 204800000 [ 8626.268265] Lustre: DEBUG MARKER: STOP OST3 [ 8632.982434] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8633.262151] Lustre: lustre-OST0003: new disk, initializing [ 8633.270537] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8633.281958] Lustre: 206829:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8635.255940] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8635.477934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8635.489400] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8635.567097] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8637.296587] LustreError: 206989:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8637.990094] Lustre: Failing over lustre-OST0003 [ 8638.159281] Lustre: server umount lustre-OST0003 complete [ 8640.493540] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8640.501254] LustreError: 205042:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8640.509145] Lustre: Skipped 1 previous similar message [ 8640.540852] LustreError: 205042:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 8642.160308] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8651.799434] Lustre: DEBUG MARKER: START OST3 again [ 8659.697779] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8660.087312] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8660.108615] Lustre: Skipped 1 previous similar message [ 8660.124070] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8661.117631] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8662.091181] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8662.094359] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8667.657169] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8689.828299] Lustre: Failing over lustre-OST0003 [ 8689.972722] Lustre: server umount lustre-OST0003 complete [ 8691.175564] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8691.202709] LustreError: 206355:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8691.205908] Lustre: Skipped 1 previous similar message [ 8691.240830] LustreError: 206355:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 8711.653646] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8711.668118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8711.685085] Lustre: Skipped 2 previous similar messages [ 8711.708732] Lustre: Skipped 1 previous similar message [ 8715.700057] Lustre: server umount lustre-MDT0000 complete [ 8720.312803] LustreError: 202490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787889395 with bad export cookie 12717151302503569902 [ 8720.313451] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8720.330107] LustreError: 202490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8726.997868] Lustre: server umount lustre-MDT0001 complete [ 8732.783679] Lustre: server umount lustre-OST0000 complete [ 8737.275048] Lustre: server umount lustre-OST0001 complete [ 8745.447118] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 8756.567593] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8807.081800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8807.350379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8807.395177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8807.500239] Lustre: lustre-MDT0000: new disk, initializing [ 8807.605049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8807.626970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8812.954230] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8825.837293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8826.023715] Lustre: 212288:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8826.107234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8826.113896] Lustre: Skipped 1 previous similar message [ 8826.206856] Lustre: lustre-MDT0001: new disk, initializing [ 8826.377237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8826.394710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8831.533416] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8841.335535] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8847.193215] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8853.802334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8854.036691] Lustre: 214259:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8856.083920] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8856.220179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8860.673544] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 8870.736668] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8875.676400] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8875.888966] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8880.368941] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8880.626994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8886.764844] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8886.766929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8886.783826] Lustre: Skipped 3 previous similar messages [ 8886.791123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8886.808458] Lustre: Skipped 5 previous similar messages [ 8888.704725] Lustre: server umount lustre-OST0000 complete [ 8899.516461] Lustre: server umount lustre-MDT0000 complete [ 8902.115228] LustreError: 212299:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8902.136920] LustreError: 212299:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 17 previous similar messages [ 8904.825572] LustreError: 212279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787889579 with bad export cookie 12717151302503572198 [ 8904.832782] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8904.844874] LustreError: 212279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8905.739034] Lustre: server umount lustre-MDT0001 complete [ 8910.879049] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 23:59:42 (1787889582) [ 8931.698843] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 8944.264670] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 8988.207725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8988.522356] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8988.585724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8988.593563] Lustre: Skipped 1 previous similar message [ 8988.706703] Lustre: lustre-MDT0000: new disk, initializing [ 8988.711627] Lustre: Skipped 1 previous similar message [ 8988.846144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8988.851892] Lustre: Skipped 1 previous similar message [ 8993.222544] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9003.390853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9003.525541] Lustre: 218885:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9003.788233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9008.093400] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9017.161678] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9022.619927] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9028.833368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9029.082494] Lustre: 220856:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9030.813505] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9034.479251] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9043.718736] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9048.139706] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9048.379491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9052.658346] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9052.891249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9056.227184] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9056.236767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9056.248717] Lustre: Skipped 2 previous similar messages [ 9056.254234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9056.262291] Lustre: Skipped 2 previous similar messages [ 9069.023224] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9069.325848] Lustre: server umount lustre-OST0000 complete [ 9075.682487] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9079.460535] Lustre: server umount lustre-MDT0000 complete [ 9082.338878] LustreError: 222197:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9082.375661] LustreError: 222197:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9083.717557] LustreError: 218879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787889758 with bad export cookie 12717151302503573080 [ 9083.744606] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9084.091137] Lustre: server umount lustre-MDT0001 complete [ 9094.417033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9095.099242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9095.114754] Lustre: Skipped 5 previous similar messages [ 9100.243932] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9110.327424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9115.452285] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9123.542283] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9129.550742] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9137.458250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9144.380542] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9152.405885] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9165.794974] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9165.798824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9165.811210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9165.822298] Lustre: Skipped 4 previous similar messages [ 9165.838568] Lustre: Skipped 9 previous similar messages [ 9169.091462] Lustre: server umount lustre-OST0000 complete [ 9179.366825] Lustre: server umount lustre-MDT0000 complete [ 9182.884848] LustreError: 223057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787889857 with bad export cookie 12717151302503573962 [ 9182.895539] LustreError: 223057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9182.897263] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9183.373654] Lustre: server umount lustre-MDT0001 complete [ 9293.136834] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [ 9296.052119] Key type lgssc unregistered [ 9296.421549] LNet: 226743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9296.428476] LNetError: 226743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9296.453672] LNet: Removed LNI 192.168.204.131@tcp [ 9297.225144] Key type .llcrypt unregistered [ 9297.229054] Key type ._llcrypt unregistered [ 9324.298111] Key type ._llcrypt registered [ 9324.300900] Key type .llcrypt registered [ 9324.414958] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 9343.114634] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 9344.170499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9344.203895] alg: No test for adler32 (adler32-zlib) [ 9345.234251] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [ 9345.594756] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [ 9347.271438] Key type lgssc registered [ 9348.508692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9388.886384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9388.913341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9390.297831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9390.365145] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9390.492954] Lustre: lustre-MDT0000: new disk, initializing [ 9390.610944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9390.632229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9395.518386] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9407.694139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9407.842704] Lustre: 230101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9407.887732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9407.891838] Lustre: Skipped 1 previous similar message [ 9407.993448] Lustre: lustre-MDT0001: new disk, initializing [ 9408.102039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9408.138453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9408.149162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9412.522113] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9424.838868] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9435.796132] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9447.028564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9447.587293] Lustre: lustre-OST0000: new disk, initializing [ 9447.593700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9447.610090] Lustre: 232070:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9447.846956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9453.710948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9453.728172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9453.964648] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9456.532674] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9469.206371] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9476.444920] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9476.905229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9484.448837] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9484.985391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9489.896568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9489.898783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9489.927547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9495.013481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9495.025960] Lustre: Skipped 1 previous similar message [ 9500.129298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9500.154741] Lustre: Skipped 1 previous similar message [ 9503.199127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9503.464231] Lustre: server umount lustre-OST0000 complete [ 9510.370253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9510.382079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9510.395293] Lustre: Skipped 1 previous similar message [ 9510.412039] LustreError: Skipped 1 previous similar message [ 9510.454741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9510.466761] Lustre: Skipped 2 previous similar messages [ 9515.490183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9515.496716] Lustre: Skipped 1 previous similar message [ 9516.015917] Lustre: server umount lustre-MDT0000 complete [ 9520.290860] LustreError: 230093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890195 with bad export cookie 3389017898891537799 [ 9520.295098] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9520.313494] LustreError: 230093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9520.611755] Lustre: server umount lustre-MDT0001 complete [ 9543.063869] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 9557.176946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9557.880572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9563.104937] LustreError: 235007:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9563.123876] LustreError: 235007:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 9563.651620] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9567.217212] LustreError: 235008:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9572.321881] LustreError: 235007:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9573.856830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9574.186904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9579.987619] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9583.849469] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9591.085748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9591.403380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9598.741656] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9608.469630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9608.656896] Lustre: lustre-OST0001: new disk, initializing [ 9608.659472] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9608.664701] Lustre: 237318:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9608.746338] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9615.925381] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9616.535385] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9616.558662] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9616.676423] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9628.356979] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9638.342242] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9654.775827] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 00:12:06 (1787890326) [ 9657.833523] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9657.843769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9657.844992] Lustre: Skipped 2 previous similar messages [ 9657.849131] Lustre: Skipped 2 previous similar messages [ 9663.527304] Lustre: server umount lustre-MDT0000 complete [ 9667.663758] LustreError: 234989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890342 with bad export cookie 3389017898891538681 [ 9667.666941] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9667.672147] LustreError: 234989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9668.024554] Lustre: server umount lustre-MDT0001 complete [ 9683.295613] Lustre: 228322:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787890342/real 1787890342] req@ffff89fbc2130000 x1874738565734912/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787890358 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9683.328096] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9684.583933] Lustre: server umount lustre-OST0000 complete [ 9687.200235] Lustre: 228323:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787890346/real 1787890346] req@ffff89fbc2132940 x1874738565735168/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787890362 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9688.481811] Lustre: 228322:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787890347/real 1787890347] req@ffff89fcfca0fc00 x1874738565735424/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787890363 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9688.802470] Lustre: server umount lustre-OST0001 complete [ 9695.850482] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [ 9707.177965] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [ 9745.428866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9745.634770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9745.683290] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9745.752680] Lustre: lustre-MDT0000: new disk, initializing [ 9745.823330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9745.841509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9750.468817] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9763.139642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9763.278243] Lustre: 241261:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9763.347463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9763.351079] Lustre: Skipped 1 previous similar message [ 9763.436500] Lustre: lustre-MDT0001: new disk, initializing [ 9763.541776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9763.550166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9768.837935] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9778.466368] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9784.804458] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9792.467423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9792.955554] Lustre: lustre-OST0000: new disk, initializing [ 9792.958982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9792.969035] Lustre: 243230:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9794.162928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9794.175953] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9794.349189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9800.099436] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9810.789378] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9815.738615] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9815.984477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9821.103085] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9821.369855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9825.250673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9825.275677] Lustre: Skipped 1 previous similar message [ 9825.283619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9825.291070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9825.305992] Lustre: Skipped 5 previous similar messages [ 9829.535695] Lustre: server umount lustre-OST0000 complete [ 9835.491567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9835.493358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9835.531207] Lustre: Skipped 1 previous similar message [ 9840.971326] Lustre: server umount lustre-MDT0000 complete [ 9845.731740] LustreError: 241273:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9845.767587] LustreError: 241273:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9846.018224] LustreError: 241254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890520 with bad export cookie 3389017898891540816 [ 9846.032818] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9846.043262] LustreError: 241254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9847.009843] Lustre: server umount lustre-MDT0001 complete [ 9859.438501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9860.126338] LustreError: 245449:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9860.210608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9860.216470] Lustre: Skipped 2 previous similar messages [ 9866.087119] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9870.818158] LustreError: 245450:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9870.841526] LustreError: 245450:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 9876.922287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9883.497199] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9892.854792] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9898.844867] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9906.610337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9915.434373] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [ 9924.993689] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10071.018602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10071.028635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10071.052756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10071.064843] Lustre: Skipped 4 previous similar messages [10085.343139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10085.511615] Lustre: server umount lustre-OST0000 complete [10092.516431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10095.528739] Lustre: server umount lustre-MDT0000 complete [10098.663591] LustreError: 246179:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10098.711906] LustreError: 246179:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [10099.704062] LustreError: 245431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787890774 with bad export cookie 3389017898891541698 [10099.722680] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10099.728179] LustreError: 245431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10100.239543] Lustre: server umount lustre-MDT0001 complete [10194.641482] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [10197.875230] Key type lgssc unregistered [10198.225568] LNet: 249062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10198.237599] LNetError: 249062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10198.254456] LNet: Removed LNI 192.168.204.131@tcp [10199.288190] Key type .llcrypt unregistered [10199.294364] Key type ._llcrypt unregistered [10225.807850] Key type ._llcrypt registered [10225.811030] Key type .llcrypt registered [10225.924244] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [10242.886943] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [10243.821131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10243.886109] alg: No test for adler32 (adler32-zlib) [10244.973449] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [10245.172173] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [10246.823188] Key type lgssc registered [10247.847560] Lustre: Echo OBD driver; http://www.lustre.org/ [10299.173802] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [10309.958177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10309.982698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10311.373768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10311.437362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10311.595976] Lustre: lustre-MDT0000: new disk, initializing [10311.801880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10311.839345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10316.716787] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10330.265697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10330.558869] Lustre: 253135:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10330.642649] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10330.648477] Lustre: Skipped 1 previous similar message [10330.750312] Lustre: lustre-MDT0001: new disk, initializing [10330.849383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10330.890774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10330.910213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10336.166048] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10341.711952] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10348.364081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10348.629568] Lustre: lustre-OST0000: new disk, initializing [10348.638360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10348.653123] Lustre: 254767:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10348.832860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10352.629209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10352.638573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10352.753165] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10355.932465] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10367.406850] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10367.671842] Lustre: lustre-OST0001: new disk, initializing [10367.687878] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10367.700667] Lustre: 255638:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10367.852960] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10374.028305] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10377.829125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10377.839987] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10377.949659] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10385.817178] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10393.202717] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10407.905232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10407.912656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10407.928722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10408.929091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10408.931016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10408.945219] Lustre: Skipped 2 previous similar messages [10408.966094] Lustre: Skipped 3 previous similar messages [10412.758635] Lustre: server umount lustre-MDT0000 complete [10417.089337] LustreError: 253127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787891091 with bad export cookie 6299502256384618661 [10417.097854] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10417.600538] Lustre: server umount lustre-MDT0001 complete [10432.035965] Lustre: server umount lustre-OST0000 complete [10435.551175] Lustre: 250639:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787891094/real 1787891094] req@ffff89fbc20b0f00 x1874739509405056/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787891110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10435.587292] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10436.129154] Lustre: server umount lustre-OST0001 complete [10451.205651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10461.369445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10469.493219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10479.512522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10502.660301] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [10512.671206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10512.688674] Lustre: lustre-MDT0000: reset Object Index mappings [10513.058825] LustreError: 260648:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10513.079885] LustreError: 260648:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [10513.160148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10518.228068] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10518.499129] LustreError: 260649:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10522.614310] LustreError: 260648:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10526.929347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10526.943977] Lustre: lustre-MDT0001: reset Object Index mappings [10527.473616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10531.577457] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10534.685840] Lustre: 261791:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10542.089914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10549.503697] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10552.810554] LustreError: 262661:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10557.950980] LustreError: 262661:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10557.994556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10559.812105] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10560.263074] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10560.267889] Lustre: Skipped 1 previous similar message [10565.661021] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10568.498524] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10578.731660] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10583.278807] Lustre: 263663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10590.149640] Lustre: server umount lustre-MDT0000 complete [10591.207527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10591.238821] LustreError: 261711:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10591.269196] LustreError: 261711:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [10594.785731] LustreError: 260630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787891269 with bad export cookie 6299502256384625451 [10594.790707] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10594.796971] LustreError: 260630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10595.229633] Lustre: server umount lustre-MDT0001 complete [10609.444101] Lustre: server umount lustre-OST0000 complete [10612.703156] Lustre: 250639:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787891271/real 1787891271] req@ffff89fbc404e940 x1874739509455488/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787891287 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10612.731787] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10612.739765] Lustre: Skipped 3 previous similar messages [10613.877663] Lustre: server umount lustre-OST0001 complete [10636.037240] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 00:28:27 (1787891307) [10655.769560] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [10669.088360] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [10711.371914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10711.669664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10711.694540] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10711.779186] Lustre: lustre-MDT0000: new disk, initializing [10711.885825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10711.916977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10717.701233] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10729.963876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10730.090294] Lustre: 267922:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10730.145310] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10730.149234] Lustre: Skipped 1 previous similar message [10730.213632] Lustre: lustre-MDT0001: new disk, initializing [10730.323050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10730.335796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10734.890951] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10745.685713] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10751.736893] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10758.226291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10758.453904] Lustre: lustre-OST0000: new disk, initializing [10758.459505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10758.464777] Lustre: 269892:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10760.276685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10760.300248] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10760.419726] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10765.033287] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10776.310168] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10782.109943] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10782.497840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10788.554671] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10788.894581] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10791.395324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10791.396149] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10791.404398] LustreError: Skipped 1 previous similar message [10791.405352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10791.433989] Lustre: Skipped 1 previous similar message [10796.514221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10796.526360] Lustre: Skipped 2 previous similar messages [10797.563940] Lustre: server umount lustre-OST0000 complete [10801.647657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10807.833192] Lustre: server umount lustre-MDT0000 complete [10811.850400] LustreError: 267915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787891486 with bad export cookie 6299502256384627544 [10811.851105] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10811.860019] LustreError: 267915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10812.146395] Lustre: server umount lustre-MDT0001 complete [10832.228361] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [10843.085530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10843.522493] LustreError: 272826:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10843.564949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10843.570156] Lustre: Skipped 2 previous similar messages [10847.837323] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10856.454228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10862.129028] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10865.068811] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10872.574076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10879.583503] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10887.847359] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10887.974448] Lustre: lustre-OST0001: new disk, initializing [10887.977647] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10887.982833] Lustre: 275133:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10889.584974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10889.592610] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10889.683239] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10893.627255] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [10902.747190] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10906.636041] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10912.466355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10912.495705] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10920.419377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10920.426241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10920.442214] Lustre: Skipped 4 previous similar messages [10920.456031] Lustre: Skipped 6 previous similar messages [10934.239524] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10934.506119] Lustre: server umount lustre-MDT0000 complete [10935.787466] LustreError: 272826:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10935.811366] LustreError: 272826:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [10938.765830] LustreError: 272806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787891613 with bad export cookie 6299502256384628426 [10938.773851] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10938.780391] LustreError: 272806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10939.122581] Lustre: server umount lustre-MDT0001 complete [10954.119911] Lustre: server umount lustre-OST0000 complete [10956.255655] Lustre: 250640:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787891615/real 1787891615] req@ffff89fbc23ca580 x1874739509561088/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787891631 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10956.295810] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10956.312087] Lustre: Skipped 1 previous similar message [10958.891972] Lustre: server umount lustre-OST0001 complete [10978.666992] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [10990.492263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10991.033140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10991.038789] Lustre: Skipped 3 previous similar messages [10995.948532] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11000.322953] LustreError: 278190:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11000.386735] LustreError: 278190:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [11005.948351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11012.010326] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11016.516163] Lustre: 279330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [11024.929467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11032.969091] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11035.740848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [11043.336972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11048.970385] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [11050.815694] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11059.632283] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11063.897878] Lustre: 281210:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [11069.732386] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [11069.918473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [11081.537493] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 00:35:53 (1787891753) [11098.596499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11098.609300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11098.635604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11098.642326] Lustre: Skipped 8 previous similar messages [11104.425429] Lustre: server umount lustre-MDT0000 complete [11108.369428] LustreError: 278171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787891783 with bad export cookie 6299502256384634768 [11108.371551] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11108.378955] LustreError: 278171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11114.872677] Lustre: server umount lustre-MDT0001 complete [11125.117378] Lustre: server umount lustre-OST0000 complete [11129.384435] Lustre: server umount lustre-OST0001 complete [11137.347452] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [11147.646593] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [11199.258295] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [11212.921547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11213.327842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11213.411320] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11213.549938] Lustre: lustre-MDT0000: new disk, initializing [11213.748674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11218.346976] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11229.124983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11229.251653] Lustre: 285454:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11229.559542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11234.314576] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11246.688102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11247.040889] Lustre: 287088:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11247.170971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11247.177561] Lustre: Skipped 5 previous similar messages [11248.714865] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11253.692737] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11265.425692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11265.660700] Lustre: 287959:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11267.009652] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11267.022300] Lustre: Skipped 1 previous similar message [11267.202665] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11273.342441] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11283.665070] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11287.961251] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11287.967943] Lustre: Skipped 1 previous similar message [11306.515819] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:39:38 (1787891978) [11310.924213] Lustre: server umount lustre-MDT0000 complete [11311.584083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11311.599869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11311.619088] Lustre: Skipped 5 previous similar messages [11311.632942] LustreError: 285466:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11311.649211] LustreError: 285466:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 14 previous similar messages [11315.310759] LustreError: 285448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787891990 with bad export cookie 6299502256384642650 [11315.313347] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11315.327459] LustreError: 285448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11315.829072] Lustre: server umount lustre-MDT0001 complete [11331.145649] Lustre: server umount lustre-OST0000 complete [11335.008651] Lustre: 250639:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787891993/real 1787891993] req@ffff89fcc4080000 x1874739509721344/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787892009 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [11335.845911] Lustre: server umount lustre-OST0001 complete [11345.402510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11350.919914] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11360.492800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11365.519577] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11373.890469] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11380.355970] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11387.245255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11394.345645] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11402.859352] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11424.529246] loop1: detected capacity change from 0 to 204800000 [11424.822881] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11424.844697] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11424.864052] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11431.290889] loop1: detected capacity change from 0 to 204800000 [11437.358933] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11437.471882] LustreError: 290195:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs17152: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11437.486772] LustreError: 290195:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs17152 [11437.497475] LustreError: 290195:0:(mgs_handler.c:517:mgs_target_reg()) fs17152-MDT0001: Error checking target index: rc = -13 [11437.506635] LustreError: fs17152-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11437.512737] LustreError: 293925:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11437.519451] LustreError: 293925:0:(tgt_mount.c:2129:server_put_super()) no obd fs17152-MDT0001 [11437.526499] LustreError: 293925:0:(tgt_mount.c:116:server_deregister_mount()) fs17152-MDT0001 not registered [11437.555780] Lustre: server umount fs17152-MDT0001 complete [11437.560111] LustreError: 293925:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11447.671791] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11447.759257] Lustre: Setting parameter fs17152-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs17152-MDT0001 [11447.800852] Lustre: srv-fs17152-MDT0001: No data found on store. Initialize space. [11447.805639] Lustre: Skipped 4 previous similar messages [11447.915829] Lustre: fs17152-MDT0001: new disk, initializing [11447.921840] Lustre: Skipped 3 previous similar messages [11448.102932] Lustre: srv-fs17152-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11448.118161] Lustre: 294392:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17152-MDT0001-mdtlov: can't create new llog: rc = -115 [11448.134039] LustreError: 294392:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17152-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [11452.481702] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11459.075301] Lustre: server umount fs17152-MDT0001 complete [11471.856864] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11473.583865] loop1: detected capacity change from 0 to 204800000 [11479.991377] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11480.364860] Lustre: 295960:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17152-MDT0001-mdtlov: can't create new llog: rc = -115 [11480.381842] LustreError: 295960:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17152-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11485.604623] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11487.222808] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11491.003541] Lustre: server umount fs17152-MDT0001 complete [11502.575942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11502.583828] Lustre: Skipped 10 previous similar messages [11507.885130] Lustre: server umount lustre-MDT0000 complete [11516.908772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11517.124110] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11522.503314] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11522.547859] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11531.331791] loop1: detected capacity change from 0 to 204800000 [11537.492864] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11538.027984] Lustre: 298409:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs17152-MDT0001-mdtlov: can't create new llog: rc = -115 [11538.043824] LustreError: 298409:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs17152-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11543.350467] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11549.019993] Lustre: server umount fs17152-MDT0001 complete [11567.745701] LustreError: 293213:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11567.780207] LustreError: 293213:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 109 previous similar messages [11578.850419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11578.863416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11578.866828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11578.866837] Lustre: Skipped 4 previous similar messages [11578.868371] Lustre: Skipped 8 previous similar messages [11581.039046] Lustre: server umount lustre-OST0000 complete [11591.647659] Lustre: server umount lustre-MDT0000 complete [11595.851444] LustreError: 290188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787892270 with bad export cookie 6299502256384645331 [11595.852824] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11595.875969] LustreError: 290188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11596.318277] Lustre: server umount lustre-MDT0001 complete [11609.196774] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [11612.007476] Key type lgssc unregistered [11612.279951] LNet: 300757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11612.304533] LNetError: 300757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11612.340211] LNet: Removed LNI 192.168.204.131@tcp [11613.553391] Key type .llcrypt unregistered [11613.555751] Key type ._llcrypt unregistered [11636.104591] Key type ._llcrypt registered [11636.111434] Key type .llcrypt registered [11636.282764] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [11637.753694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11637.794982] alg: No test for adler32 (adler32-zlib) [11638.810110] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [11639.022946] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [11640.671266] Key type lgssc registered [11641.951967] Lustre: Echo OBD driver; http://www.lustre.org/ [11651.046383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11651.085149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11653.000470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11658.780642] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11662.307799] LustreError: 302030:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11662.317823] LustreError: 302030:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11667.426036] LustreError: 302031:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11669.681144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11670.224460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11675.321243] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11684.859532] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11692.822196] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11701.349627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11701.867994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11709.410913] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11714.153431] LustreError: 303861:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11718.486412] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11719.666143] LustreError: 303863:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11725.476731] LustreError: 303862:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11725.515186] LustreError: 303862:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [11735.671735] LustreError: 303861:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11735.696329] LustreError: 303861:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [11744.591958] loop1: detected capacity change from 0 to 204800000 [11744.829834] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11744.847764] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11744.885365] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11751.663163] loop1: detected capacity change from 0 to 204800000 [11753.462618] LustreError: 304395:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11753.497273] LustreError: 304395:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 10 previous similar messages [11758.322572] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11758.581385] Lustre: fs32478-OST0000: new disk, initializing [11758.595921] Lustre: srv-fs32478-OST0000: No data found on store. Initialize space. [11758.610986] Lustre: 305769:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11758.740306] Lustre: fs32478-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11766.119354] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11776.639046] Lustre: server umount fs32478-OST0000 complete [11786.888930] LustreError: 303863:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11786.921555] LustreError: 303863:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 18 previous similar messages [11792.927623] loop1: detected capacity change from 0 to 204800000 [11794.452162] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11799.527385] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11799.661633] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11800.159155] Lustre: fs32478-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11808.035395] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11818.565550] loop2: detected capacity change from 0 to 204800000 [11819.000718] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11819.011035] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11819.019679] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11824.742553] loop2: detected capacity change from 0 to 204800000 [11830.608323] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11830.712328] LustreError: fs32478-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11830.730230] LustreError: fs32478-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11830.737973] LustreError: 308811:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [11830.743322] LustreError: 308811:0:(tgt_mount.c:2129:server_put_super()) no obd fs32478-OST0001 [11830.747835] LustreError: 308811:0:(tgt_mount.c:116:server_deregister_mount()) fs32478-OST0001 not registered [11830.781722] Lustre: server umount fs32478-OST0001 complete [11830.785056] LustreError: 308811:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11841.180685] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11841.398433] Lustre: fs32478-OST0001: new disk, initializing [11841.412408] Lustre: srv-fs32478-OST0001: No data found on store. Initialize space. [11841.430821] Lustre: 309363:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11841.507653] Lustre: fs32478-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11848.641780] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11853.434039] LustreError: 304946:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11853.467689] LustreError: 304946:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 38 previous similar messages [11858.034822] Lustre: server umount fs32478-OST0000 complete [11869.582865] Lustre: server umount fs32478-OST0001 complete [11882.196181] loop1: detected capacity change from 0 to 204800000 [11889.050691] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11889.455709] Lustre: fs32478-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11896.970882] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11907.118872] loop2: detected capacity change from 0 to 204800000 [11915.195926] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11924.403263] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [11936.094993] Lustre: server umount fs32478-OST0000 complete [11946.518318] Lustre: server umount fs32478-OST0001 complete [11948.012690] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11953.142467] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11960.299487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11960.316902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11960.328844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11963.362279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11963.367051] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11963.399449] Lustre: Skipped 1 previous similar message [11966.545842] Lustre: server umount lustre-OST0000 complete [11972.064307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11972.074997] LustreError: Skipped 1 previous similar message [11972.079846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11972.105731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11976.586912] Lustre: server umount lustre-MDT0000 complete [11980.735068] LustreError: 302012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787892655 with bad export cookie 18263515728540483776 [11980.740730] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11980.747115] LustreError: 302012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11981.193390] Lustre: server umount lustre-MDT0001 complete [11994.875629] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [11998.457527] Key type lgssc unregistered [11998.793436] LNet: 315293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11998.805287] LNetError: 315293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11998.826468] LNet: Removed LNI 192.168.204.131@tcp [11999.825298] Key type .llcrypt unregistered [11999.828174] Key type ._llcrypt unregistered [12038.138619] Key type ._llcrypt registered [12038.140891] Key type .llcrypt registered [12038.259705] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [12056.210295] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [12057.411113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12057.602559] alg: No test for adler32 (adler32-zlib) [12058.789836] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [12059.047621] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [12060.719181] Key type lgssc registered [12061.868876] Lustre: Echo OBD driver; http://www.lustre.org/ [12104.196273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12104.222614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12105.517976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12105.540037] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12105.609760] Lustre: lustre-MDT0000: new disk, initializing [12105.703175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12105.720505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12110.352918] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12120.923556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12121.012301] Lustre: 319152:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12121.057241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12121.060347] Lustre: Skipped 1 previous similar message [12121.134643] Lustre: lustre-MDT0001: new disk, initializing [12121.205227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12121.237819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12121.250061] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12126.286101] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12136.676033] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12144.272913] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12151.938590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12152.408249] Lustre: lustre-OST0000: new disk, initializing [12152.415350] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12152.433292] Lustre: 321124:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12152.591588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12159.257709] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12162.113684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12162.146078] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12162.269403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12170.714843] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12173.536796] Lustre: Mounted lustre-client - version 2.17.57_108_ga833322 [12174.750209] Lustre: Unmounted lustre-client [12188.135519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12188.141441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12188.145708] LustreError: Skipped 1 previous similar message [12188.148566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12188.168874] Lustre: Skipped 1 previous similar message [12189.157938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12189.164642] Lustre: Skipped 2 previous similar messages [12192.202057] Lustre: server umount lustre-OST0000 complete [12198.370303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12198.386515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12198.417656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12199.395293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12203.288373] Lustre: server umount lustre-MDT0000 complete [12207.779392] LustreError: 319144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787892882 with bad export cookie 8055077508952538114 [12207.780089] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12207.796417] LustreError: 319144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12208.020313] Lustre: server umount lustre-MDT0001 complete [12224.853699] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [12228.585774] Key type lgssc unregistered [12228.968552] LNet: 323314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12228.993192] LNetError: 323314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12305.832936] LNet: Removed LNI 192.168.204.131@tcp [12306.727890] Key type .llcrypt unregistered [12306.730946] Key type ._llcrypt unregistered [12324.914343] Key type ._llcrypt registered [12324.916880] Key type .llcrypt registered [12324.993701] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [12344.793930] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [12345.792555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12345.836672] alg: No test for adler32 (adler32-zlib) [12347.310464] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [12347.688192] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [12349.399213] Key type lgssc registered [12350.676533] Lustre: Echo OBD driver; http://www.lustre.org/ [12381.432966] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 00:57:33 (1787893053) [12387.887500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12387.930408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12389.181560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12389.230228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12389.322335] Lustre: lustre-MDT0000: new disk, initializing [12389.382173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12389.410521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12394.223461] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12404.536979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12404.685798] Lustre: 326537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12404.741290] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12404.745179] Lustre: Skipped 1 previous similar message [12404.833338] Lustre: lustre-MDT0001: new disk, initializing [12405.009212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12405.074394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12405.099233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12409.462857] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12418.786788] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12424.657672] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12431.052201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12431.383964] Lustre: lustre-OST0000: new disk, initializing [12431.392496] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12431.398781] Lustre: 328507:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12431.478791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12438.047696] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12438.609153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12438.616251] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12438.748137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12447.256889] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12464.609034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12464.624440] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12464.626965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12464.639329] Lustre: Skipped 1 previous similar message [12468.302493] Lustre: server umount lustre-OST0000 complete [12474.849906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12474.853046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12474.862982] Lustre: Skipped 1 previous similar message [12474.871478] Lustre: Skipped 1 previous similar message [12478.152603] Lustre: server umount lustre-MDT0000 complete [12481.962030] LustreError: 326530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787893156 with bad export cookie 18341329248260620649 [12481.974320] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12481.981030] LustreError: 326530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12482.253871] Lustre: server umount lustre-MDT0001 complete [12492.585444] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [12494.993423] Key type lgssc unregistered [12495.292286] LNet: 330419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12495.312579] LNetError: 330419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12495.333662] LNet: Removed LNI 192.168.204.131@tcp [12496.287503] Key type .llcrypt unregistered [12496.292251] Key type ._llcrypt unregistered [12519.395865] Key type ._llcrypt registered [12519.399641] Key type .llcrypt registered [12519.517637] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [12521.011077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12521.036468] alg: No test for adler32 (adler32-zlib) [12522.093136] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [12522.399508] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [12524.159175] Key type lgssc registered [12525.413581] Lustre: Echo OBD driver; http://www.lustre.org/ [12533.220812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12533.232905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12534.732206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12539.813527] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12544.992792] LustreError: 331691:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12545.023390] LustreError: 331691:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12546.019685] LustreError: 331692:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12549.047686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12549.440911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12553.357577] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12559.648072] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12564.613259] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12570.669588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12570.926518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12576.340678] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12582.870973] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12593.238501] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12594.744838] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12600.804464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12600.804520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12600.822092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12600.827666] Lustre: Skipped 1 previous similar message [12604.423658] Lustre: server umount lustre-MDT0000 complete [12605.927232] LustreError: 332420:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12607.821267] LustreError: 331673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787893282 with bad export cookie 7547093076442290627 [12607.828555] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12607.843397] LustreError: 331673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12611.039891] LustreError: 334072:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12611.042770] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12611.063527] LustreError: 334072:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [12611.089500] Lustre: Skipped 1 previous similar message [12611.096935] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12611.102263] Lustre: Skipped 3 previous similar messages [12616.316244] LustreError: 331688:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.31@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12616.316810] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [12616.325479] LustreError: 331688:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [12616.337397] Lustre: Skipped 2 previous similar messages [12621.436512] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.31@tcp (stopping) [12621.445978] Lustre: Skipped 1 previous similar message [12622.304639] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12622.581514] Lustre: server umount lustre-MDT0001 complete [12626.649250] Lustre: server umount lustre-OST0000 complete [12634.840375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12639.307320] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12641.994939] Lustre: 335887:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.131@tcp [12643.898287] Lustre: server umount MGS complete [12651.584028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12652.000191] LustreError: 336461:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12652.022491] LustreError: 336461:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [12652.085556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12656.194786] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12665.650508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12666.021358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12670.309719] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12679.183291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12679.773911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12686.643442] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12697.618769] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12699.427804] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12705.277071] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12706.976286] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12711.910498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12711.912647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12711.918016] LustreError: Skipped 1 previous similar message [12711.919712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12711.919718] Lustre: Skipped 2 previous similar messages [12711.975710] Lustre: Skipped 1 previous similar message [12713.568475] Lustre: server umount lustre-OST0000 complete [12717.536855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12717.547824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12722.145068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12722.145115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12722.152643] Lustre: Skipped 2 previous similar messages [12723.649717] Lustre: server umount lustre-MDT0000 complete [12727.266621] LustreError: 336462:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12727.304326] LustreError: 336462:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 7 previous similar messages [12727.741492] LustreError: 337918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787893402 with bad export cookie 7547093076442291467 [12727.742821] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12727.756814] LustreError: 337918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12728.087662] Lustre: server umount lustre-MDT0001 complete [12739.459403] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [12741.917822] Key type lgssc unregistered [12742.161586] LNet: 339901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12742.171803] LNetError: 339901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12742.213681] LNet: Removed LNI 192.168.204.131@tcp [12743.119752] Key type .llcrypt unregistered [12743.124073] Key type ._llcrypt unregistered [12761.250971] Key type ._llcrypt registered [12761.254596] Key type .llcrypt registered [12761.436453] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [12779.141260] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [12780.521944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12780.699555] alg: No test for adler32 (adler32-zlib) [12781.755488] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [12782.003729] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [12783.703243] Key type lgssc registered [12784.781151] Lustre: Echo OBD driver; http://www.lustre.org/ [12820.678676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12820.715204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12822.027787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12822.055766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12822.186140] Lustre: lustre-MDT0000: new disk, initializing [12822.297652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12822.311395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12826.003768] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12836.277063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12836.399828] Lustre: 343063:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12836.454327] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12836.458201] Lustre: Skipped 1 previous similar message [12836.557302] Lustre: lustre-MDT0001: new disk, initializing [12836.711917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12836.787857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12836.801753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12841.437120] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12850.962300] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12857.232128] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12864.674981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12864.970908] Lustre: lustre-OST0000: new disk, initializing [12864.976304] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12864.981872] Lustre: 345034:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12865.076734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12872.306908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12872.320974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12872.452658] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12872.461951] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [12883.372984] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12888.576390] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12888.896574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12893.973610] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12894.219249] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12898.277646] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12898.285351] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12898.296562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12898.302617] Lustre: Skipped 1 previous similar message [12902.398271] Lustre: server umount lustre-OST0000 complete [12908.528569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12908.532463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12908.539924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12908.539932] Lustre: Skipped 1 previous similar message [12908.561477] Lustre: Skipped 1 previous similar message [12913.633062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12913.643068] Lustre: Skipped 2 previous similar messages [12918.757536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12918.767994] Lustre: Skipped 1 previous similar message [12921.311167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12921.539868] Lustre: server umount lustre-MDT0000 complete [12925.759045] LustreError: 343055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787893600 with bad export cookie 4549062122027289616 [12925.767680] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12925.778901] LustreError: 343055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12926.122828] Lustre: server umount lustre-MDT0001 complete [12936.281448] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 01:06:48 (1787893608) [12953.789859] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [12964.900885] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [13004.394922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13004.663286] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13004.698592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13004.761340] Lustre: lustre-MDT0000: new disk, initializing [13004.836527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13004.867938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13009.787995] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13021.967881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13022.239949] Lustre: 349794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13022.353821] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13022.362065] Lustre: Skipped 1 previous similar message [13022.485622] Lustre: lustre-MDT0001: new disk, initializing [13022.662706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13022.736721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13022.773652] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13028.176881] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13038.340186] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13044.422338] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13051.512816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13051.771617] Lustre: lustre-OST0000: new disk, initializing [13051.777562] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13051.793215] Lustre: 351761:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13051.920798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13057.895398] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13061.718389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13061.734176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13061.867248] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13067.727154] Lustre: DEBUG MARKER: oleg431-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13103.074535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13103.082252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13103.098702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13103.133112] Lustre: Skipped 1 previous similar message [13113.325790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13113.339808] Lustre: Skipped 3 previous similar messages [13115.361196] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13115.562552] Lustre: server umount lustre-OST0000 complete [13119.970819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13119.985541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13120.007708] Lustre: Skipped 1 previous similar message [13126.072857] Lustre: server umount lustre-MDT0000 complete [13128.672236] LustreError: 349800:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13128.708234] LustreError: 349800:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [13130.361814] LustreError: 349787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787893805 with bad export cookie 4549062122027290491 [13130.370714] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13130.379818] LustreError: 349787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13130.600963] Lustre: server umount lustre-MDT0001 complete [13142.941691] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13146.018271] Key type lgssc unregistered [13146.506409] LNet: 353942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13146.514527] LNetError: 353942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13146.538512] LNet: Removed LNI 192.168.204.131@tcp [13147.424731] Key type .llcrypt unregistered [13147.426348] Key type ._llcrypt unregistered [13246.306823] Key type ._llcrypt registered [13246.312952] Key type .llcrypt registered [13246.402284] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13246.711592] Key type .llcrypt unregistered [13246.713637] Key type ._llcrypt unregistered [13337.306736] Key type ._llcrypt registered [13337.311628] Key type .llcrypt registered [13337.441888] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13337.848305] Key type .llcrypt unregistered [13337.852921] Key type ._llcrypt unregistered [13372.917623] Key type ._llcrypt registered [13372.920279] Key type .llcrypt registered [13373.099838] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13373.566310] Key type .llcrypt unregistered [13373.568235] Key type ._llcrypt unregistered [13408.825932] Key type ._llcrypt registered [13408.828241] Key type .llcrypt registered [13408.945815] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13409.241038] Key type .llcrypt unregistered [13409.248450] Key type ._llcrypt unregistered [13497.555933] Key type ._llcrypt registered [13497.557904] Key type .llcrypt registered [13497.674573] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13497.957353] Key type .llcrypt unregistered [13497.963175] Key type ._llcrypt unregistered [13538.672477] Key type ._llcrypt registered [13538.674706] Key type .llcrypt registered [13538.895612] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13539.335418] Key type .llcrypt unregistered [13539.341824] Key type ._llcrypt unregistered [13577.944268] Key type ._llcrypt registered [13577.945907] Key type .llcrypt registered [13578.037919] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13578.340387] Key type .llcrypt unregistered [13578.354877] Key type ._llcrypt unregistered [13612.128024] Key type ._llcrypt registered [13612.130415] Key type .llcrypt registered [13612.241722] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13612.493328] Key type .llcrypt unregistered [13612.499204] Key type ._llcrypt unregistered [13650.272586] Key type ._llcrypt registered [13650.280514] Key type .llcrypt registered [13650.420799] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13650.798334] Key type .llcrypt unregistered [13650.800445] Key type ._llcrypt unregistered [13688.157044] Key type ._llcrypt registered [13688.160987] Key type .llcrypt registered [13688.300926] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13688.738657] Key type .llcrypt unregistered [13688.745692] Key type ._llcrypt unregistered [13752.974661] Key type ._llcrypt registered [13752.976941] Key type .llcrypt registered [13753.062480] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13753.333418] Key type .llcrypt unregistered [13753.335500] Key type ._llcrypt unregistered [13792.147596] Key type ._llcrypt registered [13792.154667] Key type .llcrypt registered [13792.313437] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13792.699316] Key type .llcrypt unregistered [13792.701175] Key type ._llcrypt unregistered [13826.043421] Key type ._llcrypt registered [13826.046791] Key type .llcrypt registered [13826.146711] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [13826.585886] Key type .llcrypt unregistered [13826.590538] Key type ._llcrypt unregistered [13885.466639] Key type ._llcrypt registered [13885.469238] Key type .llcrypt registered [13885.628243] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [13902.160531] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [13903.208680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13903.368288] alg: No test for adler32 (adler32-zlib) [13904.443332] Lustre: Lustre: Build Version: 2.17.57_108_ga833322 [13904.710396] LNet: Added LNI 192.168.204.131@tcp [8/256/0/180] [13906.423237] Key type lgssc registered [13907.386492] Lustre: Echo OBD driver; http://www.lustre.org/ [13949.694880] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13949.732649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13951.106591] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13951.151813] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13951.277722] Lustre: tst250fs-MDT0000: new disk, initializing [13951.382209] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13951.404868] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13955.934457] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13967.131604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13967.304892] Lustre: 374391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13967.385459] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13967.393040] Lustre: Skipped 1 previous similar message [13967.512340] Lustre: tst250fs-MDT0001: new disk, initializing [13967.611894] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13967.681807] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13967.695960] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13972.641352] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13977.130880] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13983.895724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13984.276700] Lustre: tst250fs-OST0000: new disk, initializing [13984.279695] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13984.289898] Lustre: 376025:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13984.391808] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13990.586295] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [13991.011589] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13991.031949] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13991.118101] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14002.802387] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14003.045265] Lustre: tst250fs-OST0001: new disk, initializing [14003.048939] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [14003.057654] Lustre: 376895:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14003.176298] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14009.881868] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_default_debug -1 all [14009.972838] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14010.003624] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14010.162131] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14016.065244] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [14018.618445] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [14018.630636] Lustre: Skipped 1 previous similar message [14044.641943] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14044.660404] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14044.675170] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14046.177405] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [14046.181050] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14047.832623] Lustre: server umount tst250fs-MDT0000 complete [14052.808521] LustreError: 374383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787894727 with bad export cookie 14126704162243061613 [14052.809211] LustreError: MGC192.168.204.131@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14052.820305] LustreError: 374383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14053.232952] Lustre: server umount tst250fs-MDT0001 complete [14068.076586] Lustre: server umount tst250fs-OST0000 complete [14082.204853] Lustre: server umount tst250fs-OST0001 complete [14100.903413] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [14108.470787] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [14140.587431] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 01:26:53 (1787894813) [14142.044081] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [14158.744819] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing set_hostid [14167.805027] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing load_modules_local [14202.385280] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13894 sec ========= 01:27:54 (1787894874) [14204.296585] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:27:56 (1787894876) === [14208.942451] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:28:00 (1787894880) === [14252.758426] Lustre: DEBUG MARKER: oleg431-server.virtnet: executing unload_modules_local [14255.599921] Key type lgssc unregistered [14255.893143] LNet: 386043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14255.909226] LNetError: 386043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14256.934395] LNet: Removed LNI 192.168.204.131@tcp [14257.883521] Key type .llcrypt unregistered [14257.886258] Key type ._llcrypt unregistered