[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 508652748 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2432MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002378] x2apic enabled [ 0.003013] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007388] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008031] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010179] LSM: Security Framework initializing [ 0.011079] Yama: becoming mindful. [ 0.012063] SELinux: Initializing. [ 0.013071] *** VALIDATE selinux *** [ 0.021338] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025560] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027105] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029146] *** VALIDATE tmpfs *** [ 0.031215] *** VALIDATE proc *** [ 0.032303] *** VALIDATE cgroup *** [ 0.033017] *** VALIDATE cgroup2 *** [ 0.035002] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036192] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038036] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042632] debug: unmapping init [mem 0xffffffff8a459000-0xffffffff8a460fff] [ 0.045211] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046796] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047031] ... version: 2 [ 0.048019] ... bit width: 48 [ 0.049015] ... generic registers: 4 [ 0.050016] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052021] ... fixed-purpose events: 3 [ 0.053016] ... event mask: 000000070000000f [ 0.054343] rcu: Hierarchical SRCU implementation. [ 0.056768] smp: Bringing up secondary CPUs ... [ 0.057619] x86: Booting SMP configuration: [ 0.058037] .... node #0, CPUs: #1 #2 #3 [ 0.061762] smp: Brought up 1 node, 4 CPUs [ 0.063018] smpboot: Max logical packages: 1 [ 0.064024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.242567] node 0 deferred pages initialised in 176ms [ 0.246141] devtmpfs: initialized [ 0.247300] x86/mm: Memory block size: 128MB [ 0.249820] gcov: version magic: 0x41383552 [ 0.251270] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.256192] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.260694] pinctrl core: initialized pinctrl subsystem [ 0.263417] [ 0.265012] ************************************************************* [ 0.268017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.272018] ** ** [ 0.275016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.279017] ** ** [ 0.282015] ** This means that this kernel is built to expose internal ** [ 0.285018] ** IOMMU data structures, which may compromise security on ** [ 0.288024] ** your system. ** [ 0.291016] ** ** [ 0.295019] ** If you see this message and you are not debugging the ** [ 0.297012] ** kernel, report this immediately to your vendor! ** [ 0.301015] ** ** [ 0.303013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.306017] ************************************************************* [ 0.310841] NET: Registered protocol family 16 [ 0.312471] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.315093] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.317129] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.320127] cpuidle: using governor menu [ 0.321816] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.322926] PCI: Using configuration type 1 for base access [ 0.323166] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.331182] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.334023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.338065] cryptd: max_cpu_qlen set to 1000 [ 0.341701] ACPI: Added _OSI(Module Device) [ 0.343018] ACPI: Added _OSI(Processor Device) [ 0.345010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.346015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.351397] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.358025] ACPI: Interpreter enabled [ 0.360067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.362038] ACPI: Using IOAPIC for interrupt routing [ 0.364171] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.367708] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.379424] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.381068] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.385026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.388114] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.394675] acpiphp: Slot [2] registered [ 0.395353] acpiphp: Slot [5] registered [ 0.398205] acpiphp: Slot [6] registered [ 0.399181] acpiphp: Slot [7] registered [ 0.401206] acpiphp: Slot [8] registered [ 0.403184] acpiphp: Slot [9] registered [ 0.404284] acpiphp: Slot [10] registered [ 0.406195] acpiphp: Slot [3] registered [ 0.407122] acpiphp: Slot [4] registered [ 0.409144] acpiphp: Slot [11] registered [ 0.411145] acpiphp: Slot [12] registered [ 0.413182] acpiphp: Slot [13] registered [ 0.414174] acpiphp: Slot [14] registered [ 0.416123] acpiphp: Slot [15] registered [ 0.417189] acpiphp: Slot [16] registered [ 0.419115] acpiphp: Slot [17] registered [ 0.420198] acpiphp: Slot [18] registered [ 0.422125] acpiphp: Slot [19] registered [ 0.423163] acpiphp: Slot [20] registered [ 0.425095] acpiphp: Slot [21] registered [ 0.427162] acpiphp: Slot [22] registered [ 0.429174] acpiphp: Slot [23] registered [ 0.430129] acpiphp: Slot [24] registered [ 0.432150] acpiphp: Slot [25] registered [ 0.434130] acpiphp: Slot [26] registered [ 0.437202] acpiphp: Slot [27] registered [ 0.438100] acpiphp: Slot [28] registered [ 0.440172] acpiphp: Slot [29] registered [ 0.441124] acpiphp: Slot [30] registered [ 0.443164] acpiphp: Slot [31] registered [ 0.445081] PCI host bridge to bus 0000:00 [ 0.446026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.448028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.450032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.453035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.456042] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.459041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.461359] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.465228] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.468571] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.479864] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.484958] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.487027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.490034] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.492025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.494642] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.496995] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.500057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.502826] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.509018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.523021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.529018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.535781] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.543080] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.550030] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.566035] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.575738] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.581023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.586015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.603020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.610969] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.622022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.630022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.651039] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.666928] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.671015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.676020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.689018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.695154] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.702022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.705017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.712017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.719911] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.727022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.735057] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.747025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.755000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.756044] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.760634] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.764795] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.767198] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.773117] iommu: Default domain type: Passthrough [ 0.774000] SCSI subsystem initialized [ 0.774197] ACPI: bus type USB registered [ 0.775000] usbcore: registered new interface driver usbfs [ 0.777168] usbcore: registered new interface driver hub [ 0.779100] usbcore: registered new device driver usb [ 0.781173] pps_core: LinuxPPS API ver. 1 registered [ 0.783019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.786188] PTP clock support registered [ 0.789106] EDAC MC: Ver: 3.0.0 [ 0.791086] PCI: Using ACPI for IRQ routing [ 0.793015] NetLabel: Initializing [ 0.794013] NetLabel: domain hash size = 128 [ 0.795012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.796143] NetLabel: unlabeled traffic allowed by default [ 0.798176] vgaarb: loaded [ 0.800332] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.802026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.808126] clocksource: Switched to clocksource kvm-clock [ 0.931923] VFS: Disk quotas dquot_6.6.0 [ 0.933679] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.936763] *** VALIDATE ramfs *** [ 0.938241] *** VALIDATE hugetlbfs *** [ 0.939781] pnp: PnP ACPI init [ 0.943219] pnp: PnP ACPI: found 6 devices [ 0.960051] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.962493] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.963832] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.965417] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.966894] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.968389] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.970069] NET: Registered protocol family 2 [ 0.971592] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.975045] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.977536] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.981386] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.983962] TCP: Hash tables configured (established 65536 bind 65536) [ 0.985971] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.987927] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.989800] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.991872] NET: Registered protocol family 1 [ 0.994046] RPC: Registered named UNIX socket transport module. [ 0.996107] RPC: Registered udp transport module. [ 0.998053] RPC: Registered tcp transport module. [ 0.999290] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.000995] NET: Registered protocol family 44 [ 1.001968] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.003302] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.004554] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.006882] PCI: CLS 0 bytes, default 64 [ 1.008587] Unpacking initramfs... [ 2.444849] debug: unmapping init [mem 0xffffa0acbcc54000-0xffffa0acbffbffff] [ 2.451650] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.454134] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.458288] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.972753] Initialise system trusted keyrings [ 2.974623] Key type blacklist registered [ 2.976690] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.986503] zbud: loaded [ 2.989874] *** VALIDATE nfs *** [ 2.991335] *** VALIDATE nfs4 *** [ 2.993120] pstore: using deflate compression [ 2.996932] Platform Keyring initialized [ 3.153348] NET: Registered protocol family 38 [ 3.155685] Key type asymmetric registered [ 3.157913] Asymmetric key parser 'x509' registered [ 3.160556] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.164531] io scheduler mq-deadline registered [ 3.167074] io scheduler kyber registered [ 3.169365] io scheduler bfq registered [ 3.171696] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.175489] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.179239] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.183268] ACPI: Power Button [PWRF] [ 3.190564] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.199689] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.217815] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.225748] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.244725] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.275995] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.307305] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.329246] Non-volatile memory driver v1.3 [ 3.331028] Linux agpgart interface v0.103 [ 3.372790] virtio_blk virtio1: [vda] 136536 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.376306] vda: detected capacity change from 0 to 69906432 [ 3.393185] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.396599] vdb: detected capacity change from 0 to 1073741824 [ 3.415764] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.419077] vdc: detected capacity change from 0 to 2621440000 [ 3.435510] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.439252] vdd: detected capacity change from 0 to 2621440000 [ 3.455143] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.458682] vde: detected capacity change from 0 to 4294967296 [ 3.476197] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.479408] vdf: detected capacity change from 0 to 4294967296 [ 3.486642] libphy: Fixed MDIO Bus: probed [ 3.502271] usbcore: registered new interface driver usbserial_generic [ 3.507297] usbserial: USB Serial support registered for generic [ 3.511747] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.518685] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.521294] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.525746] mousedev: PS/2 mouse device common for all mice [ 3.528832] rtc_cmos 00:05: RTC can wake from S4 [ 3.533845] rtc_cmos 00:05: registered as rtc0 [ 3.535295] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.536468] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.540669] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.542516] intel_pstate: CPU model not supported [ 3.547608] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.551259] hid: raw HID events driver (C) Jiri Kosina [ 3.553597] usbcore: registered new interface driver usbhid [ 3.556563] usbhid: USB HID core driver [ 3.558405] drop_monitor: Initializing network drop monitor service [ 3.561172] Initializing XFRM netlink socket [ 3.563494] NET: Registered protocol family 10 [ 3.566669] Segment Routing with IPv6 [ 3.568551] NET: Registered protocol family 17 [ 3.571260] mpls_gso: MPLS GSO support [ 3.579295] RAS: Correctable Errors collector initialized. [ 3.581529] AVX version of gcm_enc/dec engaged. [ 3.583023] AES CTR mode by8 optimization enabled [ 3.671450] sched_clock: Marking stable (3671417368, 0)->(4657009925, -985592557) [ 3.676136] registered taskstats version 1 [ 3.678631] Loading compiled-in X.509 certificates [ 3.680977] zswap: loaded using pool lzo/zbud [ 3.709911] Key type big_key registered [ 3.726239] Key type encrypted registered [ 3.728298] ima: No TPM chip found, activating TPM-bypass! [ 3.730831] ima: Allocated hash algorithm: sha1 [ 3.733033] ima: No architecture policies found [ 3.735253] evm: Initialising EVM extended attributes: [ 3.737566] evm: security.selinux [ 3.739209] evm: security.ima [ 3.740544] evm: security.capability [ 3.742042] evm: HMAC attrs: 0x1 [ 3.745741] rtc_cmos 00:05: setting system clock to 2026-06-12 00:57:07 UTC (1781225827) [ 3.753747] debug: unmapping init [mem 0xffffffff8b403000-0xffffffff8b5fffff] [ 3.758378] debug: unmapping init [mem 0xffffffff8a182000-0xffffffff8a458fff] [ 3.768087] Write protecting the kernel read-only data: 28672k [ 3.772416] debug: unmapping init [mem 0xffffffff88803000-0xffffffff889fffff] [ 3.776541] debug: unmapping init [mem 0xffffffff89114000-0xffffffff891fffff] [ 3.820112] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.832603] systemd[1]: Detected virtualization kvm. [ 3.835207] systemd[1]: Detected architecture x86-64. [ 3.837817] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.866218] systemd[1]: No hostname configured. [ 3.868161] systemd[1]: Set hostname to . [ 3.870718] random: systemd: uninitialized urandom read (16 bytes read) [ 3.873967] systemd[1]: Initializing machine ID from random generator. [ 4.064834] random: systemd: uninitialized urandom read (16 bytes read) [ 4.068120] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.072528] random: systemd: uninitialized urandom read (16 bytes read) [ 4.075750] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.080437] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.814126] device-mapper: uevent: version 1.0.3 [ 4.816347] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.637299] virtio_net virtio0 ens2: renamed from eth0 [ 5.667254] random: fast init done [ 5.726663] scsi host0: ata_piix [ 5.738105] scsi host1: ata_piix [ 5.739771] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.742517] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.263521] random: crng init done [ 10.266717] random: 7 urandom warning(s) missed due to ratelimiting [ 10.276925] dracut-initqueue[577]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 12.566304] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.119225] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.079977] SELinux: Disabled at runtime. [ 17.193725] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.210676] systemd[1]: Detected virtualization kvm. [ 17.213464] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.918010] hrtimer: interrupt took 3918961 ns [ 19.165696] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.182497] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.193764] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.214770] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.229766] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.289669] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.325614] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ 20.085515] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Journal Service. [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 20.916437] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.410743] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.478541] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.384319] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.503821] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (10s / no limit)[ 29.778064] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (11s / no limit)[ 30.906852] NFS: Registering the id_resolver key type [ 30.913394] Key type id_resolver registered [ 30.917375] Key type id_legacy registered [ **] A start job is running for Configur…only root support (12s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg223-server login: [ 112.522501] libcfs: loading out-of-tree module taints kernel. [ 112.609567] Key type ._llcrypt registered [ 112.611372] Key type .llcrypt registered [ 112.745410] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 135.928603] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 138.079885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 138.103982] alg: No test for adler32 (adler32-zlib) [ 139.723342] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 140.561747] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 142.328299] Key type lgssc registered [ 145.484131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 167.301258] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 222.936556] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 237.468924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 237.596375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 239.307176] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 239.414544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 239.648557] Lustre: lustre-MDT0000: new disk, initializing [ 239.798423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 239.829222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 246.566910] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 264.206389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 264.411789] Lustre: 6547:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 264.471958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 264.482307] Lustre: Skipped 1 previous similar message [ 264.672816] Lustre: lustre-MDT0001: new disk, initializing [ 264.784179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 264.823135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 264.840667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 270.381163] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 276.104300] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 289.104863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 289.606189] Lustre: lustre-OST0000: new disk, initializing [ 289.617631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 289.627800] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 289.688329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 295.990720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 296.007337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 296.158060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 297.358939] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 312.552239] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 312.709416] Lustre: lustre-OST0001: new disk, initializing [ 312.713931] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 312.724371] Lustre: 9508:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 312.809150] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 319.868692] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 322.621284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 322.628683] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 322.675052] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 337.135645] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 347.532265] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 355.925926] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing check_logdir /tmp/testlogs/ [ 362.865509] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing yml_node [ 371.037163] Lustre: DEBUG MARKER: Client: 2.17.53.68 [ 374.225267] Lustre: DEBUG MARKER: MDS: 2.17.53.68 [ 377.626348] Lustre: DEBUG MARKER: OSS: 2.17.53.68 [ 379.903992] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 11 21:03:21 EDT 2026 [ 402.865281] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 405.143934] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 423.396381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 423.401070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 423.408175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 424.931716] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 424.947534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 424.965077] Lustre: Skipped 2 previous similar messages [ 424.981630] Lustre: Skipped 3 previous similar messages [ 429.207868] Lustre: server umount lustre-MDT0000 complete [ 433.709348] LustreError: 6539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226257 with bad export cookie 7561039959497469245 [ 433.714903] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 433.725421] LustreError: 6539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 435.168692] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 435.172358] LustreError: 6553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 435.172574] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 435.197079] Lustre: Skipped 1 previous similar message [ 435.255746] LustreError: 6553:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 438.180141] LustreError: 6555:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 438.180639] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 438.222101] Lustre: Skipped 2 previous similar messages [ 440.317190] Lustre: server umount lustre-MDT0001 complete [ 444.926323] Lustre: server umount lustre-OST0000 complete [ 450.101448] Lustre: server umount lustre-OST0001 complete [ 459.318358] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 472.448744] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 514.526482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 514.832219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 514.859625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 514.949542] Lustre: lustre-MDT0000: new disk, initializing [ 515.027692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 515.050889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 520.315325] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 532.289620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 532.548190] Lustre: 15031:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 532.633333] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 532.643716] Lustre: Skipped 1 previous similar message [ 532.859451] Lustre: lustre-MDT0001: new disk, initializing [ 533.047208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 533.122628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 533.133158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 539.452274] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 547.808897] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 552.461801] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 560.825698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 561.482566] Lustre: lustre-OST0000: new disk, initializing [ 561.486799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 561.500527] Lustre: 16772:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 561.728683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 563.558413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 563.568514] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 563.699806] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 569.906477] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 580.315260] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 587.076703] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 587.298611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 593.383826] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 593.773720] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 599.521910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 599.527497] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 599.558809] LustreError: Skipped 1 previous similar message [ 599.589305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 609.764200] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 609.779939] Lustre: Skipped 4 previous similar messages [ 610.271137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 610.614166] Lustre: server umount lustre-OST0000 complete [ 620.000262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 620.002763] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.029676] Lustre: Skipped 2 previous similar messages [ 621.514954] Lustre: server umount lustre-MDT0000 complete [ 625.122594] LustreError: 15037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 625.148464] LustreError: 15037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 625.618811] LustreError: 15022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226449 with bad export cookie 7561039959497470778 [ 625.632778] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 625.998786] Lustre: server umount lustre-MDT0001 complete [ 632.413805] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 21:07:34 (1781226454) [ 660.079848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 661.042790] LustreError: 19505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 661.298587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 666.599708] LustreError: 19506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 667.409699] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 675.808574] LustreError: 19506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 675.836984] LustreError: 19506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 678.030038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 684.648572] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 691.900225] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 696.882813] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 699.466814] Lustre: Failing over lustre-MDT0000 [ 699.921417] Lustre: server umount lustre-MDT0000 complete [ 703.603889] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 703.620504] LustreError: 19505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 703.645106] LustreError: 19505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 719.329235] Lustre: 3660:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781226527/real 1781226527] req@ffffa0ac032f2580 x1867750796605952/t0(0) o400->MGC192.168.202.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781226543 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 719.355649] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 721.171398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 730.597375] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x68ee35c8766178c0 [ 730.608995] Lustre: MGC192.168.202.123@tcp: Connection restored to 0@lo (at 0@lo) [ 731.015069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 731.018041] Lustre: Skipped 1 previous similar message [ 731.179723] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 736.169270] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 736.238354] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 736.244661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 736.314455] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 746.785882] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 752.123361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 752.127362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 752.180187] Lustre: Skipped 1 previous similar message [ 752.196228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 752.198782] Lustre: Skipped 2 previous similar messages [ 758.570488] Lustre: server umount lustre-MDT0000 complete [ 759.278301] LustreError: 19500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 759.322741] LustreError: 19500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 762.746598] LustreError: 19486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226586 with bad export cookie 7561039959497472192 [ 762.766442] LustreError: 19486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 762.777540] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 763.234811] Lustre: server umount lustre-MDT0001 complete [ 772.787130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 778.592569] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 788.630976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 794.471761] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 801.081473] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 805.858093] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 808.236440] Lustre: Failing over lustre-MDT0000 [ 809.442038] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 809.451152] Lustre: Skipped 2 previous similar messages [ 810.540333] Lustre: server umount lustre-MDT0000 complete [ 823.778269] LustreError: 22507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 823.809514] LustreError: 22507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 825.823631] Lustre: 3658:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781226633/real 1781226633] req@ffffa0ad12a4e940 x1867750796635008/t0(0) o400->MGC192.168.202.123@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781226649 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 825.840807] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 833.627119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 835.807349] LustreError: 24271:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 835.824929] LustreError: 24271:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@ffffa0ad2fa643c0 x1867750796636928/t0(0) o250->MGC192.168.202.123@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1781226659 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 835.873630] LustreError: 24271:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 836.069169] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x68ee35c876617c6a [ 836.092151] Lustre: MGC192.168.202.123@tcp: Connection restored to 0@lo (at 0@lo) [ 836.094879] Lustre: Skipped 1 previous similar message [ 836.548525] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 841.700197] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 841.705912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 841.724860] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 842.147391] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 851.028453] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid [ 857.080111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 857.084824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 857.093421] Lustre: Skipped 3 previous similar messages [ 861.963971] Lustre: server umount lustre-MDT0000 complete [ 866.803712] LustreError: 22486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226690 with bad export cookie 7561039959497473130 [ 866.821137] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 866.839184] LustreError: 22486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 867.654709] Lustre: server umount lustre-MDT0001 complete [ 881.075539] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 21:11:41 (1781226701) [ 902.724687] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 915.480380] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 962.595804] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 975.322990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 975.551175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 975.571488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 975.637534] Lustre: lustre-MDT0000: new disk, initializing [ 975.757581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 975.761176] Lustre: Skipped 3 previous similar messages [ 975.794928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 980.493622] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 992.529474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 992.687773] Lustre: 28494:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 993.080732] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 998.876086] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1008.871546] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1014.064544] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1022.425603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1022.744347] Lustre: 30237:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1024.827921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1024.918246] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 1025.001647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1025.030225] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST [ 1030.586693] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1038.920201] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1049.614255] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 1081.324709] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1081.340358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.343360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1081.371980] Lustre: Skipped 2 previous similar messages [ 1081.372443] Lustre: Skipped 3 previous similar messages [ 1083.684529] Lustre: server umount lustre-OST0000 complete [ 1094.935189] Lustre: server umount lustre-MDT0000 complete [ 1096.684227] LustreError: 28505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1096.703425] LustreError: 28505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 1099.357342] LustreError: 28485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226923 with bad export cookie 7561039959497473536 [ 1099.360724] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1099.370896] LustreError: 28485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1099.787875] Lustre: server umount lustre-MDT0001 complete [ 1112.734456] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 1115.546340] Key type lgssc unregistered [ 1115.852047] LNet: 32088:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.858532] LNetError: 32088:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.879123] LNet: Removed LNI 192.168.202.123@tcp [ 1117.064192] Key type .llcrypt unregistered [ 1117.065830] Key type ._llcrypt unregistered [ 1153.365545] Key type ._llcrypt registered [ 1153.370388] Key type .llcrypt registered [ 1153.517524] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 1154.164546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1154.188246] alg: No test for adler32 (adler32-zlib) [ 1155.259311] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 1155.470708] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 1157.136403] Key type lgssc registered [ 1158.683959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1168.453638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1168.489782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1170.206396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1175.802529] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1179.628298] LustreError: 33487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1179.664033] LustreError: 33487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1184.738169] LustreError: 33488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1186.230269] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1186.713466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1192.532600] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1199.837143] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1204.383142] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1212.769562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1213.191966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1220.222785] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1227.379352] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1231.747448] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:747 to 0x280000402:833) [ 1246.954029] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 21:17:48 (1781227068) [ 1249.310476] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1251.534307] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1261.321917] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 21:18:03 (1781227083) [ 1274.655240] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 21:18:15 (1781227095) [ 1276.552841] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1287.638165] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1287.642641] Lustre: Skipped 2 previous similar messages [ 1290.709226] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 21:18:31 (1781227111) [ 1293.281814] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 1293.287976] Lustre: Skipped 2 previous similar messages [ 1303.267696] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1303.271913] Lustre: Skipped 4 previous similar messages [ 1320.420514] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 21:19:02 (1781227142) [ 1323.779921] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1327.823719] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1339.280531] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 21:19:20 (1781227160) [ 1341.757312] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1341.771384] Lustre: Skipped 6 previous similar messages [ 1359.089769] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 21:19:40 (1781227180) [ 1379.296423] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1379.313199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1379.336723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1380.325314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1380.326345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1380.327223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1380.373312] Lustre: Skipped 1 previous similar message [ 1383.556982] Lustre: server umount lustre-OST0000 complete [ 1390.570304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.592787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1390.604216] Lustre: Skipped 1 previous similar message [ 1394.997480] Lustre: server umount lustre-MDT0000 complete [ 1395.681586] LustreError: 37450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1395.695375] LustreError: 37450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1399.870307] LustreError: 34205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227223 with bad export cookie 5542559871244211537 [ 1399.889219] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1399.903471] LustreError: 34205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1400.394857] Lustre: server umount lustre-MDT0001 complete [ 1415.374596] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 1418.833777] Key type lgssc unregistered [ 1419.255749] LNet: 40996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1419.263140] LNetError: 40996:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1419.276846] LNet: Removed LNI 192.168.202.123@tcp [ 1420.261447] Key type .llcrypt unregistered [ 1420.263685] Key type ._llcrypt unregistered [ 1441.918269] Key type ._llcrypt registered [ 1441.922271] Key type .llcrypt registered [ 1442.101728] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 1460.145624] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 1462.026743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1462.038235] alg: No test for adler32 (adler32-zlib) [ 1463.127278] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 1463.487348] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 1465.295228] Key type lgssc registered [ 1466.697451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1508.365657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1508.413790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1509.783831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1509.815682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1509.944830] Lustre: lustre-MDT0000: new disk, initializing [ 1510.073727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1510.107681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1514.893842] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1527.580460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1527.762367] Lustre: 44119:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1527.853189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1527.862399] Lustre: Skipped 1 previous similar message [ 1527.953205] Lustre: lustre-MDT0001: new disk, initializing [ 1528.117263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1528.187209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1528.204677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1533.435173] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1542.202242] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1546.522781] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1554.269306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1554.618473] Lustre: lustre-OST0000: new disk, initializing [ 1554.630918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1554.661352] Lustre: 45859:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1554.820675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.236848] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1563.261778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1563.283164] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1563.484057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1572.346040] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1580.322461] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1580.663520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1587.041242] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1587.346318] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1594.337142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1594.339162] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1594.344445] LustreError: Skipped 1 previous similar message [ 1594.369537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1594.376255] Lustre: Skipped 1 previous similar message [ 1594.397375] Lustre: Skipped 1 previous similar message [ 1597.014719] Lustre: server umount lustre-OST0000 complete [ 1604.577659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1604.583325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1604.601143] Lustre: Skipped 1 previous similar message [ 1604.623109] Lustre: Skipped 1 previous similar message [ 1608.683526] Lustre: server umount lustre-MDT0000 complete [ 1612.427908] LustreError: 44109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227436 with bad export cookie 8689310738905000580 [ 1612.451817] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1613.063859] Lustre: server umount lustre-MDT0001 complete [ 1623.524758] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 21:24:04 (1781227444) [ 1631.029948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1631.418991] LustreError: 48175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1631.442838] LustreError: 48175:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1631.495883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1636.833302] LustreError: 48176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1636.848522] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1641.952887] LustreError: 48175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1647.073981] LustreError: 48176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1647.834806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1648.342422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1653.879195] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1660.060447] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1664.980972] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1673.429846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1673.889832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1681.746570] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1687.927723] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1696.393658] LustreError: 50365:0:(llog.c:444:llog_init_handle()) llog type is not specified! [ 1697.365856] Lustre: Modifying parameter general.timeout=1 in log params [ 1699.416859] Lustre: Modifying parameter general.timeout=3 in log params [ 1699.427862] Lustre: Skipped 1 previous similar message [ 1701.605580] Lustre: Modifying parameter general.timeout=5 in log params [ 1701.611764] Lustre: Skipped 1 previous similar message [ 1705.731939] Lustre: Modifying parameter general.timeout=9 in log params [ 1705.736175] Lustre: Skipped 3 previous similar messages [ 1714.557891] Lustre: Modifying parameter general.timeout=17 in log params [ 1714.563424] Lustre: Skipped 7 previous similar messages [ 1731.557952] Lustre: Modifying parameter general.timeout=32 in log params [ 1731.568884] Lustre: Skipped 14 previous similar messages [ 1764.156292] Lustre: Modifying parameter general.timeout=64 in log params [ 1764.161430] Lustre: Skipped 31 previous similar messages [ 1835.444460] Lustre: Disabling parameter general.timeout= in log params [ 1835.457143] Lustre: Skipped 65 previous similar messages [ 1838.604137] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 21:27:40 (1781227660) [ 1841.939266] Lustre: 57315:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1844.134261] Lustre: 57414:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1848.309438] Lustre: 57623:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1848.317873] Lustre: 57623:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1852.919977] Lustre: 57836:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= change rule4 rate=5000 (mode = 2) failed: rc = -17 [ 1852.932801] Lustre: 57836:0:(mgs_llog.c:1437:mgs_modify_param()) Skipped 1 previous similar message [ 1882.705966] Lustre: 59192:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1882.744499] LustreError: 59192:0:(mgs_handler.c:1202:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1886.867362] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 21:28:28 (1781227708) [ 1904.609655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1904.626101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1904.660736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1907.243709] Lustre: server umount lustre-MDT0000 complete [ 1910.261161] LustreError: 48170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1910.297510] LustreError: 48170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1911.691634] LustreError: 59536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227735 with bad export cookie 8689310738905001462 [ 1911.712521] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1911.726423] LustreError: 59536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1912.547418] Lustre: server umount lustre-MDT0001 complete [ 1929.071187] Lustre: server umount lustre-OST0000 complete [ 1964.101975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.468531] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1964.534556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1964.550976] Lustre: Skipped 22 previous similar messages [ 1964.935732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.391614] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1979.723878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.809857] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1979.825343] Lustre: 61726:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1985.675882] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 1992.319572] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1996.199806] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2004.272023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2004.411680] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2004.658827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2004.669382] Lustre: Skipped 1 previous similar message [ 2011.154573] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2017.456822] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2019.338590] Lustre: 63954:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 2030.568825] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2030.573366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2030.574861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2030.574868] Lustre: Skipped 3 previous similar messages [ 2030.631761] Lustre: Skipped 3 previous similar messages [ 2032.116326] Lustre: server umount lustre-OST0000 complete [ 2036.710735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2036.725068] LustreError: Skipped 1 previous similar message [ 2036.729717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2036.747069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2036.752420] Lustre: Skipped 1 previous similar message [ 2043.175906] Lustre: server umount lustre-MDT0000 complete [ 2045.929993] LustreError: 61740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2045.953597] LustreError: 61740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2047.349634] LustreError: 61721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227871 with bad export cookie 8689310738905004619 [ 2047.361614] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2047.370496] LustreError: 61721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2047.840323] Lustre: server umount lustre-MDT0001 complete [ 2060.876747] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 2063.694596] Key type lgssc unregistered [ 2064.024525] LNet: 65145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2064.029803] LNetError: 65145:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2064.043988] LNet: Removed LNI 192.168.202.123@tcp [ 2064.872445] Key type .llcrypt unregistered [ 2064.875122] Key type ._llcrypt unregistered [ 2091.340287] Key type ._llcrypt registered [ 2091.343822] Key type .llcrypt registered [ 2091.472598] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 2093.061040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2093.099205] alg: No test for adler32 (adler32-zlib) [ 2094.195619] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 2094.418559] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 2096.056849] Key type lgssc registered [ 2097.528205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2106.578100] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2106.601918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.357120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2113.163622] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2117.609538] LustreError: 66544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.627991] LustreError: 66544:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2122.156864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.662235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2127.225576] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2132.617155] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2136.234247] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2143.185133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2143.770238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2150.742984] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2156.533607] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2166.530621] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 2178.530167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.535169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2178.561906] Lustre: Skipped 2 previous similar messages [ 2178.586851] Lustre: Skipped 2 previous similar messages [ 2183.664203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2183.673565] Lustre: Skipped 2 previous similar messages [ 2184.625798] Lustre: server umount lustre-MDT0000 complete [ 2188.794272] LustreError: 66545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.814213] LustreError: 66545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2189.367320] LustreError: 66524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228013 with bad export cookie 10775147643209802181 [ 2189.369063] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2189.375249] LustreError: 66524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2189.769670] Lustre: server umount lustre-MDT0001 complete [ 2205.060869] Lustre: server umount lustre-OST0000 complete [ 2235.077235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2235.275346] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2235.302298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2235.742961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2240.804342] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2251.993962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2252.077578] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2252.110595] Lustre: 71077:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2252.760710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2257.253832] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2264.171422] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2268.067624] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2276.328290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2276.546142] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2277.094532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2285.833516] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2293.077149] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2295.827339] Lustre: 73316:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2298.140120] Lustre: Modifying parameter general.jobid_var=disable in log params [ 2306.536532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2306.549744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2306.578993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2307.040304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2312.168473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2312.176218] Lustre: Skipped 2 previous similar messages [ 2317.279545] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2317.287593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2317.299054] Lustre: Skipped 1 previous similar message [ 2317.554432] Lustre: server umount lustre-OST0000 complete [ 2322.402223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2322.422443] Lustre: Skipped 1 previous similar message [ 2327.523044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2327.532582] Lustre: Skipped 4 previous similar messages [ 2328.679592] Lustre: server umount lustre-MDT0000 complete [ 2332.647352] LustreError: 71084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.667156] LustreError: 71084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2332.674348] LustreError: 71067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228156 with bad export cookie 10775147643209803028 [ 2332.686295] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2332.689385] LustreError: 71067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2332.981113] Lustre: server umount lustre-MDT0001 complete [ 2345.574327] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 2349.044812] Key type lgssc unregistered [ 2349.394932] LNet: 74511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2349.404029] LNetError: 74511:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2349.418662] LNet: Removed LNI 192.168.202.123@tcp [ 2350.286687] Key type .llcrypt unregistered [ 2350.289645] Key type ._llcrypt unregistered [ 2380.558870] Key type ._llcrypt registered [ 2380.561346] Key type .llcrypt registered [ 2380.720161] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 2382.559111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2382.624619] alg: No test for adler32 (adler32-zlib) [ 2383.752435] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 2384.120298] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 2385.888608] Key type lgssc registered [ 2387.266710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2395.245755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2395.264125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.794707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2400.954828] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2406.371386] LustreError: 75910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.401869] LustreError: 75910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2410.734088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2411.585503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2417.005936] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2424.676795] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2428.409304] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2437.252545] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2438.149447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2444.809723] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2452.226759] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2462.569545] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2463.564685] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2465.544944] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2465.552961] Lustre: Skipped 1 previous similar message [ 2467.754679] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2467.770728] Lustre: Skipped 1 previous similar message [ 2471.878329] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2471.883802] Lustre: Skipped 3 previous similar messages [ 2480.557923] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2480.560282] Lustre: Skipped 8 previous similar messages [ 2497.373274] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2497.376263] Lustre: Skipped 14 previous similar messages [ 2529.376053] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2529.381564] Lustre: Skipped 28 previous similar messages [ 2578.323403] Lustre: 83683:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2580.951638] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 21:40:03 (1781228403) [ 2582.429731] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2584.142174] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 21:40:06 (1781228406) [ 2590.695657] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2590.703528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.722558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2593.767692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2593.778450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2593.780794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.793745] Lustre: Skipped 1 previous similar message [ 2598.892407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2598.899323] Lustre: Skipped 1 previous similar message [ 2602.975551] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2603.191587] Lustre: server umount lustre-OST0000 complete [ 2609.122202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2609.134142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2609.144472] Lustre: Skipped 1 previous similar message [ 2613.951931] Lustre: server umount lustre-MDT0000 complete [ 2614.240955] LustreError: 75909:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2614.254738] LustreError: 75909:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2618.716791] LustreError: 78157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228442 with bad export cookie 4024988862908803934 [ 2618.719829] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2618.732836] LustreError: 78157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2619.117307] Lustre: server umount lustre-MDT0001 complete [ 2634.253033] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 2637.305560] Key type lgssc unregistered [ 2637.660497] LNet: 84919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2637.670711] LNetError: 84919:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2637.685200] LNet: Removed LNI 192.168.202.123@tcp [ 2638.517974] Key type .llcrypt unregistered [ 2638.523848] Key type ._llcrypt unregistered [ 2643.728809] Key type ._llcrypt registered [ 2643.730339] Key type .llcrypt registered [ 2643.857152] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2649.378191] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules [ 2650.859215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2650.874495] alg: No test for adler32 (adler32-zlib) [ 2651.858453] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 2652.046945] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 2653.736439] Key type lgssc registered [ 2655.316862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2656.724329] LustreError: 86089:0:(lu_object.c:1482:lu_context_key_register()) cfs_race id 60d sleeping [ 2677.254592] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 2679.889791] LustreError: 86089:0:(lu_object.c:1482:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2688.828360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2688.849106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2690.466623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2696.049685] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2700.773633] LustreError: 87097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.825419] LustreError: 87097:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2705.891083] LustreError: 87098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2707.937660] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 21:42:09 (1781228529) [ 2711.016262] LustreError: 87097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2713.978907] LustreError: 87129:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2714.017818] LustreError: 87129:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2714.028655] LustreError: 87129:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 2714.797777] Lustre: server umount lustre-MDT0000 complete [ 2729.472096] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 2732.148086] Key type lgssc unregistered [ 2732.443715] LNet: 88589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2732.449432] LNetError: 88589:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2732.461776] LNet: Removed LNI 192.168.202.123@tcp [ 2733.158799] Key type .llcrypt unregistered [ 2733.165050] Key type ._llcrypt unregistered [ 2753.291737] Key type ._llcrypt registered [ 2753.294857] Key type .llcrypt registered [ 2753.430966] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 2755.035465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2755.071767] alg: No test for adler32 (adler32-zlib) [ 2756.487890] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 2756.795079] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 2758.519110] Key type lgssc registered [ 2759.966877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2770.949901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2770.989397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2772.934196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2777.391581] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2783.203725] LustreError: 89791:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2783.244293] LustreError: 89791:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2788.323593] LustreError: 89790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2788.493130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2789.402036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2794.414879] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2800.868748] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2805.642197] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2812.940660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2813.590678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2820.274341] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2826.168153] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2876.929639] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 21:44:57 (1781228697) [ 2879.665656] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2882.489320] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 21:45:03 (1781228703) [ 2889.699563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2889.703544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2889.738616] Lustre: Skipped 2 previous similar messages [ 2889.755871] Lustre: Skipped 2 previous similar messages [ 2894.291231] Lustre: server umount lustre-MDT0000 complete [ 2894.817450] LustreError: 89790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.839974] LustreError: 89790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2898.699945] LustreError: 89771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228722 with bad export cookie 11397780856917467920 [ 2898.714427] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2898.716244] LustreError: 89771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2898.974188] Lustre: server umount lustre-MDT0001 complete [ 2913.766715] Lustre: server umount lustre-OST0000 complete [ 2927.459833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.864941] LustreError: 93725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2928.065209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2933.208231] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2933.220459] LustreError: 93726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.232432] LustreError: 93726:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2942.447633] LustreError: 93725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.465834] LustreError: 93725:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2942.967307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2943.682701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2949.458610] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 2956.876320] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2961.597839] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2976.091926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2976.446239] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2976.454331] LustreError: 93715:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2976.494941] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2976.501761] LustreError: 95593:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2976.507803] LustreError: 95593:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2976.516663] LustreError: 95593:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2976.655117] Lustre: server umount lustre-OST0000 complete [ 2976.657723] LustreError: 95593:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2988.344034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2988.475545] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2988.478477] LustreError: 93714:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2988.496521] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2988.502196] LustreError: 96057:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -98 [ 2988.504396] LustreError: 96057:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 2988.508306] LustreError: 96057:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2988.604696] Lustre: server umount lustre-OST0000 complete [ 2988.611276] LustreError: 96057:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 3001.341890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3001.556074] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3001.564402] Lustre: Found index 0 for lustre-OST0000, updating log [ 3001.570376] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 3001.659324] Lustre: lustre-OST0000: new disk, initializing [ 3001.662884] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3001.668696] Lustre: 96593:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3002.047242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3008.315891] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3012.946807] Lustre: Failing over lustre-OST0000 [ 3013.196694] Lustre: server umount lustre-OST0000 complete [ 3017.231966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3017.264719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3017.277695] Lustre: Skipped 1 previous similar message [ 3022.309539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3022.316248] Lustre: Skipped 1 previous similar message [ 3023.807728] Lustre: server umount lustre-MDT0000 complete [ 3027.431387] LustreError: 93720:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3027.488201] LustreError: 93720:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3029.511066] LustreError: 93708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781228853 with bad export cookie 11397780856917470664 [ 3029.541689] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3030.308698] Lustre: server umount lustre-MDT0001 complete [ 3041.284864] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 21:47:42 (1781228862) [ 3061.574369] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 3074.572232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.041166] LustreError: 98905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3075.329613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.133742] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3092.936799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3099.837365] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3103.758389] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 3111.911663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3113.002707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3113.014207] Lustre: Skipped 1 previous similar message [ 3120.821495] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3132.386781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3132.519769] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3132.604668] Lustre: lustre-OST0001: new disk, initializing [ 3132.620381] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3132.632779] Lustre: 101172:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3140.809022] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3142.329403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3142.340917] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3142.476637] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3150.898341] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3157.549258] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3163.115162] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3163.120309] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3163.130631] Lustre: Skipped 2 previous similar messages [ 3163.145873] Lustre: Skipped 2 previous similar messages [ 3168.231339] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3168.246722] Lustre: Skipped 3 previous similar messages [ 3169.119801] Lustre: server umount lustre-MDT0001 complete [ 3170.051576] LustreError: 101642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3170.085964] LustreError: 101642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3181.391389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3181.546634] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3181.554758] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3181.592237] Lustre: 98894:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3182.510641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3182.521845] Lustre: Skipped 1 previous similar message [ 3183.164125] LustreError: 102698:0:(obd_mount.c:225:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 3183.169825] LustreError: 102698:0:(tgt_mount.c:668:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 3183.178700] LustreError: 98898:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 3183.263396] LustreError: 102707:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.123@tcp: cfg command failed: rc = -17 [ 3183.282134] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 3183.282134] [ 3187.356925] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3187.732588] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3200.355637] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 21:50:21 (1781229021) [ 3224.671941] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 3246.031461] Lustre: 104767:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3270.471168] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3274.749902] Lustre: 105901:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3376.625384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3376.637944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3376.656130] Lustre: Skipped 1 previous similar message [ 3376.661534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3380.764504] Lustre: server umount lustre-MDT0000 complete [ 3382.245098] LustreError: 101642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3382.270722] LustreError: 101642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 3385.505084] LustreError: 98886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229209 with bad export cookie 11397780856917471658 [ 3385.521440] LustreError: 98886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3385.521595] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3386.098711] Lustre: server umount lustre-MDT0001 complete [ 3401.219345] Lustre: server umount lustre-OST0000 complete [ 3404.129537] Lustre: 89102:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781229211/real 1781229211] req@ffffa0ad12ab4780 x1867753539880064/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781229227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 3404.174085] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3404.204522] Lustre: Skipped 3 previous similar messages [ 3405.820345] Lustre: server umount lustre-OST0001 complete [ 3413.053362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3425.472466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3436.169435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3449.131242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 3475.271952] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 3487.836343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3487.862523] Lustre: lustre-MDT0000: reset Object Index mappings [ 3488.585572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3493.573311] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3503.000454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3503.022580] Lustre: lustre-MDT0001: reset Object Index mappings [ 3508.817797] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3512.190971] Lustre: 110956:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 3519.161592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3526.778253] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3528.312161] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 3533.460615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 3536.415659] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3542.539857] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 3542.540371] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 3545.565723] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3556.765358] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3562.388591] Lustre: 112799:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3616.480130] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 21:57:18 (1781229438) [ 3641.825062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3641.829044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3641.872680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3641.875755] Lustre: Skipped 4 previous similar messages [ 3648.312413] Lustre: server umount lustre-MDT0000 complete [ 3652.499720] LustreError: 109825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781229476 with bad export cookie 11397780856917518502 [ 3652.509699] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3653.093805] LustreError: 109838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3653.120061] LustreError: 109838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3658.923712] Lustre: server umount lustre-MDT0001 complete [ 3669.880132] Lustre: server umount lustre-OST0000 complete [ 3673.945072] LustreError: 113762:0:(ldlm_resource.c:1180:ldlm_resource_complain()) filter-lustre-OST0001_UUID: namespace resource [0x2c0000401:0x77:0x0].0x0 (ffffa0ac04dc1400) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3680.374513] Lustre: server umount lustre-OST0001 complete [ 3687.740156] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 3699.584698] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 3739.202316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3739.432539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3739.450594] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3739.526562] Lustre: lustre-MDT0000: new disk, initializing [ 3739.627120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3744.790797] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3752.696656] Lustre: server umount lustre-MDT0000 complete [ 3762.665568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3762.886809] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3762.894979] Lustre: 117020:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3762.903582] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3763.055539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3763.062588] Lustre: Skipped 4 previous similar messages [ 3768.026121] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3774.271236] Lustre: server umount lustre-MDT0000 complete [ 3784.341305] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 22:00:06 (1781229606) [ 3786.208978] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3788.236942] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 22:00:10 (1781229610) [ 3808.040456] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 3822.132610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3828.493461] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3839.122938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3839.293174] Lustre: 119037:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3839.360859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3839.364417] Lustre: Skipped 1 previous similar message [ 3839.442208] Lustre: lustre-MDT0001: new disk, initializing [ 3839.624469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3839.638598] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3844.900649] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3850.648743] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3858.499050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3858.807613] Lustre: lustre-OST0000: new disk, initializing [ 3858.822093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3858.834819] Lustre: 120590:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3860.487325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3860.497437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3860.677052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3864.766477] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3878.126938] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3878.355757] Lustre: lustre-OST0001: new disk, initializing [ 3878.359789] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3878.365745] Lustre: 121444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3880.420707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3880.425222] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3880.583083] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3885.721992] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 3897.356919] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3902.024571] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 4338.875964] Lustre: 127852:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 4338.888623] Lustre: 127852:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 4338.897505] Lustre: 127852:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 4338.908742] Lustre: 127852:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 4338.913706] Lustre: 127852:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 4338.921363] Lustre: 127852:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 4619.705251] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 22:14:00 (1781230440) [ 4638.179694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4638.200320] Lustre: Skipped 6 previous similar messages [ 4638.204845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4638.205161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4638.222914] Lustre: Skipped 15 previous similar messages [ 4641.589526] Lustre: server umount lustre-MDT0000 complete [ 4643.314271] LustreError: 122238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4643.350246] LustreError: 122238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4645.103230] LustreError: 119031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781230468 with bad export cookie 11397780856917586584 [ 4645.123355] LustreError: 119031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4645.125413] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4648.420681] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4648.425361] Lustre: Skipped 1 previous similar message [ 4648.429992] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4648.442927] Lustre: Skipped 1 previous similar message [ 4651.610456] Lustre: server umount lustre-MDT0001 complete [ 4661.747619] Lustre: server umount lustre-OST0000 complete [ 4670.798696] Lustre: server umount lustre-OST0001 complete [ 4678.562782] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 4690.427263] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 4731.799688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4732.118163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4732.149348] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4732.264055] Lustre: lustre-MDT0000: new disk, initializing [ 4732.441484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4732.454626] Lustre: Skipped 4 previous similar messages [ 4732.472041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4736.508634] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4747.734339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4747.921685] Lustre: 130867:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4747.977628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4747.980765] Lustre: Skipped 1 previous similar message [ 4748.053531] Lustre: lustre-MDT0001: new disk, initializing [ 4748.194564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4748.205684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4752.784757] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4760.791665] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4765.408059] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4772.752558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4773.064319] Lustre: lustre-OST0000: new disk, initializing [ 4773.070522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4773.081653] Lustre: 132611:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4774.535801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4774.550610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4774.728614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4778.740949] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 4786.398388] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4796.915991] Lustre: lustre-MDD0000: changelog on [ 4799.505693] Lustre: lustre-MDD0001: changelog on [ 6764.703762] Lustre: lustre-MDD0001: changelog off [ 6766.500088] Lustre: 131728:0:(llog_cat.c:977:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6767.273558] Lustre: lustre-MDD0000: changelog off [ 6774.762765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6774.777672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6774.791532] Lustre: Skipped 2 previous similar messages [ 6774.809575] Lustre: Skipped 3 previous similar messages [ 6778.386932] Lustre: server umount lustre-OST0000 complete [ 6784.992746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6785.003693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6785.012346] Lustre: Skipped 1 previous similar message [ 6790.115044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6790.132988] Lustre: Skipped 1 previous similar message [ 6795.234401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6795.247917] Lustre: Skipped 1 previous similar message [ 6796.255295] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6796.551553] Lustre: server umount lustre-MDT0000 complete [ 6800.167371] LustreError: 130860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232623 with bad export cookie 11397780856918365866 [ 6800.178052] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6800.181839] LustreError: 130860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6800.534174] Lustre: server umount lustre-MDT0001 complete [ 6809.890578] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 6812.743808] Key type lgssc unregistered [ 6813.263401] LNet: 141625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6813.280404] LNetError: 141625:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6813.309389] LNet: Removed LNI 192.168.202.123@tcp [ 6813.988370] Key type .llcrypt unregistered [ 6813.990719] Key type ._llcrypt unregistered [ 6834.028708] Key type ._llcrypt registered [ 6834.036908] Key type .llcrypt registered [ 6834.191408] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 6850.129397] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 6851.634933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6851.662440] alg: No test for adler32 (adler32-zlib) [ 6852.597698] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 6852.754088] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 6854.375337] Key type lgssc registered [ 6855.639145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6898.275283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6898.348363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6899.607977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6899.638633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6899.731719] Lustre: lustre-MDT0000: new disk, initializing [ 6899.812972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6899.832629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6903.662533] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6913.913168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6914.056158] Lustre: 144794:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6914.116964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6914.120835] Lustre: Skipped 1 previous similar message [ 6914.175318] Lustre: lustre-MDT0001: new disk, initializing [ 6914.266232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6914.301202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6914.315449] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6918.474341] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6926.228588] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6931.018711] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6938.362970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6938.766961] Lustre: lustre-OST0000: new disk, initializing [ 6938.772347] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6938.783792] Lustre: 146534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6938.880274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6945.537286] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 6945.852469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6945.864334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6946.058330] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6953.493565] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6960.054715] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_module obdecho/obdecho [ 6961.799321] LustreError: 147408:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6961.803300] LustreError: 147408:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6963.691842] LustreError: 147504:0:(echo_client.c:1883:echo_md_handler()) Can not resolve the path : rc = -2 [ 6975.982792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6976.001087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6976.043773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6977.003290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6977.011710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6982.115131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6982.131213] Lustre: Skipped 4 previous similar messages [ 6982.378964] Lustre: server umount lustre-MDT0000 complete [ 6986.368656] LustreError: 147759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232810 with bad export cookie 13047098592275568397 [ 6986.376574] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6986.388102] LustreError: 147759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6986.882953] Lustre: server umount lustre-MDT0001 complete [ 7000.755643] Lustre: server umount lustre-OST0000 complete [ 7023.145534] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 7035.881806] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 7070.085636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7070.258932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7070.281121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7070.348334] Lustre: lustre-MDT0000: new disk, initializing [ 7070.405573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7070.426352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7074.916116] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7085.571561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7085.718791] Lustre: 151185:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7085.763554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7085.767568] Lustre: Skipped 1 previous similar message [ 7085.833373] Lustre: lustre-MDT0001: new disk, initializing [ 7085.921581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7085.948316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7085.960265] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7090.338474] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7098.759596] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7102.601735] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7109.760651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7110.043321] Lustre: lustre-OST0000: new disk, initializing [ 7110.048339] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7110.059445] Lustre: 152927:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7110.147875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7116.525464] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7116.904239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7116.915043] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7117.039705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7124.008420] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7128.621220] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7128.902286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7135.910052] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7136.482518] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7142.889810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7142.892309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7142.902773] LustreError: Skipped 1 previous similar message [ 7142.906483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7142.946296] Lustre: Skipped 2 previous similar messages [ 7146.275214] Lustre: server umount lustre-OST0000 complete [ 7152.614383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7152.625192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7152.645562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7152.661076] Lustre: Skipped 1 previous similar message [ 7158.049732] Lustre: server umount lustre-MDT0000 complete [ 7162.754959] LustreError: 151177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781232986 with bad export cookie 13047098592275569356 [ 7162.769697] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7163.076282] Lustre: server umount lustre-MDT0001 complete [ 7166.879080] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 22:56:29 (1781232989) [ 7168.440417] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 7171.193885] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 22:56:32 (1781232992) [ 7179.156510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7179.519094] LustreError: 155194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7179.533929] LustreError: 155194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7179.643805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7183.181271] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7184.884280] LustreError: 155195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7189.986397] LustreError: 155194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7191.114549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7196.978867] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7203.260901] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7206.216815] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7213.093595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7213.475265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7213.485324] Lustre: Skipped 1 previous similar message [ 7219.401660] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7224.921320] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7235.852822] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.23@tcp (stopping) [ 7235.858726] Lustre: Skipped 2 previous similar messages [ 7236.065491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7236.079846] Lustre: Skipped 3 previous similar messages [ 7237.783860] Lustre: server umount lustre-MDT0000 complete [ 7240.949182] LustreError: 157343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7240.972539] LustreError: 157343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7241.609030] LustreError: 155175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233065 with bad export cookie 13047098592275570231 [ 7241.623459] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7241.624820] LustreError: 155175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7246.067389] LustreError: 157343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7246.115276] LustreError: 157343:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7246.310438] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7248.130529] Lustre: server umount lustre-MDT0001 complete [ 7254.912777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 7267.148173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7267.397986] LustreError: 158694:0:(llog.c:1646:llog_backup()) MGC192.168.202.123@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 7267.406191] Lustre: 158694:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.123@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 7275.833671] LustreError: 158699:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7275.853094] LustreError: 158699:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7292.149491] LustreError: 158700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7292.163085] LustreError: 158700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 7326.964729] LustreError: 158701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7326.995403] LustreError: 158701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 7328.223376] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 7328.228960] Lustre: 158694:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.202.123@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 7369.204675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7373.730537] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7374.309268] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7383.373649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7387.632350] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb5109b0b0e15b774 [ 7387.651715] Lustre: MGC192.168.202.123@tcp: Connection restored to 0@lo (at 0@lo) [ 7393.269727] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7393.841404] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7400.526573] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7402.499513] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7406.263696] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7408.107643] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7423.969664] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7423.979055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7424.001138] Lustre: Skipped 1 previous similar message [ 7424.035979] Lustre: Skipped 6 previous similar messages [ 7424.466375] Lustre: server umount lustre-MDT0000 complete [ 7428.479290] LustreError: 157635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233252 with bad export cookie 13047098592275576692 [ 7428.491352] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7428.492926] LustreError: 157635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7428.794560] Lustre: server umount lustre-MDT0001 complete [ 7443.234624] Lustre: server umount lustre-OST0000 complete [ 7466.778917] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 7477.486980] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 7520.292920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7520.665893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7520.706636] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7520.832194] Lustre: lustre-MDT0000: new disk, initializing [ 7520.973699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7520.985459] Lustre: Skipped 1 previous similar message [ 7521.013583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7525.675859] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7536.675585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7536.774690] Lustre: 163553:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7536.978739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7541.312424] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7549.103312] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7553.017327] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7560.331820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7560.623578] Lustre: lustre-OST0000: new disk, initializing [ 7560.626632] Lustre: Skipped 1 previous similar message [ 7560.633837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7560.638673] Lustre: Skipped 2 previous similar messages [ 7560.644191] Lustre: 165294:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7567.698676] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7570.538531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7570.552752] Lustre: Skipped 1 previous similar message [ 7570.558968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7570.677781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7575.311799] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7580.070480] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7580.245690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7584.687421] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7584.883204] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7591.398442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7591.403380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7591.409447] LustreError: Skipped 1 previous similar message [ 7591.410581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7591.410589] Lustre: Skipped 1 previous similar message [ 7591.455444] Lustre: Skipped 2 previous similar messages [ 7601.119636] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7601.551989] Lustre: server umount lustre-OST0000 complete [ 7608.806223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7614.159917] Lustre: server umount lustre-MDT0000 complete [ 7617.001205] LustreError: 163560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7617.038718] LustreError: 163560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 46 previous similar messages [ 7619.521307] LustreError: 163543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233443 with bad export cookie 13047098592275593534 [ 7619.525150] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7619.541640] LustreError: 163543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7619.921863] Lustre: server umount lustre-MDT0001 complete [ 7629.699945] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 23:04:11 (1781233451) [ 7636.381037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7641.819316] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7650.594301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7655.971940] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7662.892090] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7667.438665] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7675.484900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7682.127346] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7688.485434] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7706.318222] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 23:05:28 (1781233528) [ 7710.704439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7710.721996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7710.723233] Lustre: Skipped 2 previous similar messages [ 7710.725387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7710.725398] Lustre: Skipped 6 previous similar messages [ 7716.778445] Lustre: server umount lustre-OST0000 complete [ 7727.934069] Lustre: server umount lustre-MDT0000 complete [ 7732.283220] LustreError: 167542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233556 with bad export cookie 13047098592275594416 [ 7732.310884] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7732.645148] Lustre: server umount lustre-MDT0001 complete [ 7743.992464] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 7747.296607] Key type lgssc unregistered [ 7747.629154] LNet: 170951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7747.636240] LNetError: 170951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7747.668688] LNet: Removed LNI 192.168.202.123@tcp [ 7748.588900] Key type .llcrypt unregistered [ 7748.591485] Key type ._llcrypt unregistered [ 7768.366694] Key type ._llcrypt registered [ 7768.368473] Key type .llcrypt registered [ 7768.511936] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 7770.043060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7770.129877] alg: No test for adler32 (adler32-zlib) [ 7771.117106] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 7771.402424] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 7773.063534] Key type lgssc registered [ 7774.714362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7785.342990] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7785.380690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7801.195514] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7806.303266] LustreError: 172181:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.123@tcp: failed processing log, type 4: rc = -110 [ 7811.423532] LustreError: 172181:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7811.433774] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7811.443348] LustreError: MGC192.168.202.123@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7811.451717] LustreError: 172181:0:(tgt_mount.c:1840:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7811.457470] LustreError: 172181:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -22 [ 7811.463392] LustreError: 172181:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 7811.566290] Lustre: server umount lustre-OST0000 complete [ 7811.573670] LustreError: 172181:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7820.430438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7821.971726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7827.640388] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7831.530739] LustreError: 172680:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7831.587757] LustreError: 172680:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7836.656310] LustreError: 172681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7839.539413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7839.854513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7845.577322] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7851.509360] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7855.932601] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7863.518875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7863.747312] LustreError: 174267:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7863.767685] Lustre: 174267:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.202.123@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7863.941589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7871.453921] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7876.878967] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7880.685639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7880.711542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7880.758393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7882.720979] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7882.725359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7882.747063] Lustre: Skipped 1 previous similar message [ 7882.773525] Lustre: Skipped 2 previous similar messages [ 7885.776677] Lustre: server umount lustre-MDT0000 complete [ 7887.841992] LustreError: 174345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7887.877166] LustreError: 174345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7890.301323] LustreError: 172661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233714 with bad export cookie 12880463209626979205 [ 7890.303358] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7890.313557] LustreError: 172661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7890.668249] Lustre: server umount lustre-MDT0001 complete [ 7905.336454] Lustre: server umount lustre-OST0000 complete [ 7914.924939] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 7925.552047] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 7965.514893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7965.844383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7965.874749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7965.945613] Lustre: lustre-MDT0000: new disk, initializing [ 7966.031781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7966.065302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7972.080974] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7985.342437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7985.482621] Lustre: 177645:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7985.549064] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7985.555469] Lustre: Skipped 1 previous similar message [ 7985.666129] Lustre: lustre-MDT0001: new disk, initializing [ 7985.794616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7985.852408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7985.877938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7990.123703] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 7999.598922] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8003.596553] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8011.811947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8012.041804] Lustre: lustre-OST0000: new disk, initializing [ 8012.047801] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8012.053807] Lustre: 179389:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8012.157678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8018.617429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8018.650275] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8018.802616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8019.240875] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8027.376876] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8032.084695] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8032.372669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8037.358385] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8037.623887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8044.514123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8044.527112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8044.548515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8044.562052] Lustre: Skipped 1 previous similar message [ 8045.916939] Lustre: server umount lustre-OST0000 complete [ 8052.196425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8052.203934] LustreError: Skipped 1 previous similar message [ 8052.211593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8052.230100] Lustre: Skipped 1 previous similar message [ 8052.244042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8056.608695] Lustre: server umount lustre-MDT0000 complete [ 8059.888896] LustreError: 177655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8059.908773] LustreError: 177655:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8060.004418] LustreError: 177638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781233883 with bad export cookie 12880463209626979926 [ 8060.008922] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8060.014363] LustreError: 177638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8060.305387] Lustre: server umount lustre-MDT0001 complete [ 8069.579926] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 23:11:31 (1781233891) [ 8087.986574] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 8090.794297] Key type lgssc unregistered [ 8091.164733] LNet: 182173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8091.180935] LNetError: 182173:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8091.200628] LNet: Removed LNI 192.168.202.123@tcp [ 8092.101876] Key type .llcrypt unregistered [ 8092.109247] Key type ._llcrypt unregistered [ 8108.988433] Key type ._llcrypt registered [ 8108.996313] Key type .llcrypt registered [ 8109.118454] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 8110.222631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8110.294186] alg: No test for adler32 (adler32-zlib) [ 8111.407389] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 8111.719032] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 8113.351203] Key type lgssc registered [ 8114.808306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8123.044599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8123.067815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8124.605169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8129.936457] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8135.141435] LustreError: 183378:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8135.157689] LustreError: 183378:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8139.207128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8139.620452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8145.216606] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8151.764305] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8156.536497] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8165.569393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8165.867855] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8165.874119] LustreError: 185014:0:(llog.c:1687:llog_backup()) MGC192.168.202.123@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8165.888957] Lustre: 185014:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.202.123@tcp: can't backup local config lustre-OST0000: rc = -28 [ 8166.042900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8174.211412] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8180.573501] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8184.804309] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8184.814679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8184.817807] Lustre: Skipped 2 previous similar messages [ 8184.840118] Lustre: Skipped 2 previous similar messages [ 8189.302301] Lustre: server umount lustre-MDT0000 complete [ 8189.931032] LustreError: 183377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8189.971052] LustreError: 183377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8193.720494] LustreError: 183357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234017 with bad export cookie 18334982871206175103 [ 8193.722584] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8193.727531] LustreError: 183357:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8195.047078] LustreError: 183372:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8195.047674] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8195.070728] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8195.167236] LustreError: 184090:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000002b2f0d5c x1867759154653952/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8195.337242] Lustre: server umount lustre-MDT0001 complete [ 8201.574307] Lustre: server umount lustre-OST0000 complete [ 8211.308883] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 8222.214288] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 8261.743248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8261.975656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8262.012727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8262.090900] Lustre: lustre-MDT0000: new disk, initializing [ 8262.181281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8262.197201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8266.795443] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8278.941633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8279.089915] Lustre: 188392:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8279.196031] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8279.197735] Lustre: Skipped 1 previous similar message [ 8279.203735] LustreError: 189156:0:(llog.c:1687:llog_backup()) MGC192.168.202.123@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 8279.214987] LustreError: 189156:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8279.224545] Lustre: 189156:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.123@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 8279.257895] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8279.263178] Lustre: Skipped 1 previous similar message [ 8279.392345] Lustre: lustre-MDT0001: new disk, initializing [ 8279.522237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8279.570651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8279.587340] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8285.626455] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8293.292699] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8296.753237] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8303.844963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8304.196147] Lustre: *** cfs_fail_loc=131e, val=0*** [ 8304.199280] Lustre: Skipped 1 previous similar message [ 8304.203616] LustreError: 190108:0:(llog.c:1687:llog_backup()) MGC192.168.202.123@tcp: failed to backup log lustre-OST0000: rc = -28 [ 8304.214342] LustreError: 190108:0:(llog.c:1687:llog_backup()) Skipped 1 previous similar message [ 8304.219760] Lustre: 190108:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.202.123@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 8304.226990] Lustre: 190108:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 8304.268856] Lustre: lustre-OST0000: new disk, initializing [ 8304.273232] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8304.281543] Lustre: 190134:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8304.370875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8310.456198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8310.472517] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8310.588371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8311.111581] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8318.828230] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8324.347988] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8324.708988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8329.491846] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8329.684092] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8336.352655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8336.353230] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8336.385043] LustreError: Skipped 1 previous similar message [ 8336.401736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8336.410758] Lustre: Skipped 1 previous similar message [ 8336.432258] Lustre: Skipped 1 previous similar message [ 8338.172259] Lustre: server umount lustre-OST0000 complete [ 8346.080345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8346.093235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8346.123883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8348.621887] Lustre: server umount lustre-MDT0000 complete [ 8351.715655] LustreError: 188402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8351.727874] LustreError: 188402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8352.966081] LustreError: 188384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234176 with bad export cookie 18334982871206175824 [ 8352.975327] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8352.976868] LustreError: 188384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8353.337137] Lustre: server umount lustre-MDT0001 complete [ 8362.046625] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 23:16:23 (1781234183) [ 8378.912698] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 8388.698614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8389.239470] LustreError: 193095:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8389.316443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8393.422703] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8399.851642] LustreError: 193096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8399.882249] LustreError: 193096:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8401.347878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8405.478955] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8408.172815] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8414.258449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8421.834743] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8431.394511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8431.662291] Lustre: lustre-OST0001: new disk, initializing [ 8431.675616] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8431.687843] Lustre: 195352:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8431.863385] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8431.870757] Lustre: Skipped 2 previous similar messages [ 8437.937399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8437.950345] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8438.016705] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8438.458809] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8450.619847] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8462.414584] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8464.498451] Lustre: DEBUG MARKER: ADD OST3 [ 8470.077088] loop: module loaded [ 8470.194389] loop0: detected capacity change from 0 to 204800000 [ 8470.467456] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8470.480666] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8470.504940] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8477.959601] Lustre: DEBUG MARKER: STOP OST3 [ 8478.378749] loop0: detected capacity change from 0 to 204800000 [ 8484.598628] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8484.799717] Lustre: lustre-OST0003: new disk, initializing [ 8484.808171] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 8484.818689] Lustre: 197398:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8486.796729] Lustre: *** cfs_fail_loc=2109, val=2*** [ 8486.951416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 8486.971679] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 8487.028748] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 8487.599864] Lustre: Failing over lustre-OST0003 [ 8487.762738] Lustre: server umount lustre-OST0003 complete [ 8488.799813] LustreError: 197577:0:(osp_precreate.c:1281:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 8491.625459] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8491.798259] LustreError: 194547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8492.024319] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8492.031120] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8492.046023] Lustre: Skipped 2 previous similar messages [ 8499.685905] Lustre: DEBUG MARKER: START OST3 again [ 8507.459347] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8507.871990] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 8507.889670] Lustre: Skipped 1 previous similar message [ 8507.909735] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 8509.389466] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 8509.445296] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 8509.451343] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8513.167154] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8532.793635] Lustre: Failing over lustre-OST0003 [ 8532.924774] Lustre: server umount lustre-OST0003 complete [ 8535.022726] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8535.023852] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8535.031038] LustreError: Skipped 1 previous similar message [ 8535.044463] LustreError: 194547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8535.060712] Lustre: Skipped 1 previous similar message [ 8535.097818] LustreError: 194547:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 8550.368448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8550.374226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8550.378508] Lustre: Skipped 1 previous similar message [ 8551.912625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8551.932381] Lustre: Skipped 2 previous similar messages [ 8556.628975] Lustre: server umount lustre-MDT0000 complete [ 8560.062892] LustreError: 196106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234383 with bad export cookie 18334982871206176706 [ 8560.072194] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8560.077080] LustreError: 196106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8562.153096] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8562.158916] Lustre: Skipped 5 previous similar messages [ 8566.634816] Lustre: server umount lustre-MDT0001 complete [ 8571.255663] Lustre: server umount lustre-OST0000 complete [ 8575.840143] Lustre: server umount lustre-OST0001 complete [ 8583.374608] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 8594.130318] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 8634.690736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8635.127347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8635.189188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8635.289989] Lustre: lustre-MDT0000: new disk, initializing [ 8635.470177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8639.714954] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8651.282977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8651.390636] Lustre: 202729:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8651.448305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8651.451994] Lustre: Skipped 1 previous similar message [ 8651.538358] Lustre: lustre-MDT0001: new disk, initializing [ 8651.675368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8651.681852] Lustre: Skipped 1 previous similar message [ 8651.728649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8651.745909] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8656.199377] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8663.568683] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8666.946330] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8673.671137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8674.096092] Lustre: 204470:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8675.464290] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8675.564720] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8680.263283] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8687.817946] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8692.457400] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8692.754775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8697.502142] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8697.755235] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8701.413645] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8701.421547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8701.424772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8701.424780] Lustre: Skipped 2 previous similar messages [ 8701.436855] Lustre: Skipped 4 previous similar messages [ 8705.967931] Lustre: server umount lustre-OST0000 complete [ 8716.775861] LustreError: 202740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8716.798327] LustreError: 202740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 16 previous similar messages [ 8716.877585] Lustre: server umount lustre-MDT0000 complete [ 8720.894979] LustreError: 202720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234544 with bad export cookie 18334982871206179002 [ 8720.924215] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8721.237376] Lustre: server umount lustre-MDT0001 complete [ 8724.627894] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 23:22:26 (1781234546) [ 8746.557726] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 8757.694431] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 8800.597876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8800.890846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8800.926208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8800.932844] Lustre: Skipped 1 previous similar message [ 8801.029597] Lustre: lustre-MDT0000: new disk, initializing [ 8801.033851] Lustre: Skipped 1 previous similar message [ 8801.154230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8801.164268] Lustre: Skipped 1 previous similar message [ 8805.812058] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8818.759383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8819.024210] Lustre: 208899:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8819.684216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8825.875390] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8833.935296] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8839.042633] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8847.693552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8848.074747] Lustre: 210643:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8849.921179] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8854.938535] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8863.399237] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8868.529567] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8868.813494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8873.342498] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8873.517470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8875.490824] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8875.491912] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8875.504159] LustreError: Skipped 2 previous similar messages [ 8875.512548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8875.520568] Lustre: Skipped 3 previous similar messages [ 8875.529024] Lustre: Skipped 3 previous similar messages [ 8881.722851] Lustre: server umount lustre-OST0000 complete [ 8892.636950] Lustre: server umount lustre-MDT0000 complete [ 8895.971728] LustreError: 210349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8895.990382] LustreError: 210349:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8897.392144] LustreError: 208890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234721 with bad export cookie 18334982871206179884 [ 8897.403582] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8897.406889] LustreError: 208890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8898.040292] Lustre: server umount lustre-MDT0001 complete [ 8907.261985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8907.902969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8907.905784] Lustre: Skipped 4 previous similar messages [ 8913.347955] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8923.460242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8928.789735] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8935.877277] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8940.407778] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8947.881188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8956.894956] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 8964.001799] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8975.328643] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8975.339300] LustreError: Skipped 1 previous similar message [ 8975.347311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8975.354339] Lustre: Skipped 6 previous similar messages [ 8981.421310] Lustre: server umount lustre-OST0000 complete [ 8991.756287] Lustre: server umount lustre-MDT0000 complete [ 8995.059245] LustreError: 212693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781234818 with bad export cookie 18334982871206180766 [ 8995.071357] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8995.452815] Lustre: server umount lustre-MDT0001 complete [ 9103.430263] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 9107.347420] Key type lgssc unregistered [ 9107.791699] LNet: 216019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9107.797866] LNetError: 216019:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9107.828292] LNet: Removed LNI 192.168.202.123@tcp [ 9108.794413] Key type .llcrypt unregistered [ 9108.799635] Key type ._llcrypt unregistered [ 9137.468456] Key type ._llcrypt registered [ 9137.471129] Key type .llcrypt registered [ 9137.608375] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 9152.533973] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 9153.811317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9153.854077] alg: No test for adler32 (adler32-zlib) [ 9154.926697] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 9155.129840] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 9156.791216] Key type lgssc registered [ 9158.216927] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9197.203076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9197.246252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9198.570985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9198.596099] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9198.685512] Lustre: lustre-MDT0000: new disk, initializing [ 9198.764620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9198.788600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9203.164421] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9214.205550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9214.349395] Lustre: 219341:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9214.390882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9214.394495] Lustre: Skipped 1 previous similar message [ 9214.449504] Lustre: lustre-MDT0001: new disk, initializing [ 9214.536933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9214.572325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9214.583775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9218.985966] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9226.869572] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9231.748636] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9239.734677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9239.984888] Lustre: lustre-OST0000: new disk, initializing [ 9239.988555] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9239.993753] Lustre: 221081:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9240.063695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9246.606324] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9248.350066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9248.357306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9248.463749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9255.676563] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9261.429317] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9261.729180] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9266.941102] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9267.130578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9269.220485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9269.233475] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9269.240660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9269.253287] Lustre: Skipped 1 previous similar message [ 9269.278136] Lustre: Skipped 1 previous similar message [ 9274.340047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9275.102343] Lustre: server umount lustre-OST0000 complete [ 9279.456490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9279.459450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9279.480088] Lustre: Skipped 1 previous similar message [ 9279.487043] Lustre: Skipped 1 previous similar message [ 9284.576294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9284.579038] Lustre: Skipped 1 previous similar message [ 9285.420217] Lustre: server umount lustre-MDT0000 complete [ 9289.321424] LustreError: 219332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235113 with bad export cookie 9369786929976725427 [ 9289.327070] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9289.328513] LustreError: 219332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9289.535887] Lustre: server umount lustre-MDT0001 complete [ 9314.054835] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 9326.597323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9327.223388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9331.592622] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9332.705598] LustreError: 223874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9332.731807] LustreError: 223874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9336.824317] LustreError: 223875:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9341.757643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9341.920293] LustreError: 223874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9342.161177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9347.152833] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9349.822963] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9356.000291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9356.343602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9362.135521] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9372.448516] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9372.657666] Lustre: lustre-OST0001: new disk, initializing [ 9372.660862] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9372.667158] Lustre: 226133:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9372.783543] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9379.745223] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9380.455406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9380.492544] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9380.598700] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9390.487446] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9394.985155] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9408.073718] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 23:33:49 (1781235229) [ 9411.566419] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9411.599748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9411.618464] Lustre: Skipped 3 previous similar messages [ 9416.681770] LustreError: 223871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9416.710195] LustreError: 223871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9416.842501] Lustre: server umount lustre-MDT0000 complete [ 9421.536327] LustreError: 223854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235245 with bad export cookie 9369786929976726309 [ 9421.540964] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9421.550725] LustreError: 223854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9421.795709] LustreError: 223870:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9421.796988] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9421.810693] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9421.835692] LustreError: 223870:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9421.904845] Lustre: Skipped 4 previous similar messages [ 9422.283029] Lustre: server umount lustre-MDT0001 complete [ 9427.589525] Lustre: server umount lustre-OST0000 complete [ 9431.659744] Lustre: server umount lustre-OST0001 complete [ 9439.682160] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 9449.891928] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 9488.146319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9488.592825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9488.655840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9488.889407] Lustre: lustre-MDT0000: new disk, initializing [ 9489.091157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9489.140337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9494.980477] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9506.170746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9506.362627] Lustre: 230029:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9506.439545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9506.442878] Lustre: Skipped 1 previous similar message [ 9506.533352] Lustre: lustre-MDT0001: new disk, initializing [ 9506.681169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9506.699044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9511.950343] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9519.668336] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9523.362228] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9530.850648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9531.259709] Lustre: lustre-OST0000: new disk, initializing [ 9531.262805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9531.268328] Lustre: 231772:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9533.196690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9533.203914] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9533.306103] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9538.079527] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9546.209923] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9551.355485] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9551.543109] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9556.464237] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9556.717454] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9559.009454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9559.014944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9559.052525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9559.071221] Lustre: Skipped 2 previous similar messages [ 9565.231987] Lustre: server umount lustre-OST0000 complete [ 9573.351958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9573.360841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9573.382340] Lustre: Skipped 1 previous similar message [ 9576.544914] Lustre: server umount lustre-MDT0000 complete [ 9579.489250] LustreError: 230038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9579.509618] LustreError: 230038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9580.815054] LustreError: 230022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235404 with bad export cookie 9369786929976728444 [ 9580.818398] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9580.830273] LustreError: 230022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9581.390880] Lustre: server umount lustre-MDT0001 complete [ 9590.693439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9591.413047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9591.418871] Lustre: Skipped 2 previous similar messages [ 9596.302275] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9596.896451] LustreError: 233843:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9596.909374] LustreError: 233843:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9605.784700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9612.971295] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9620.965223] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9625.548468] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9634.638814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9642.575437] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [ 9648.399747] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9794.019862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9794.023396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9794.024144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9794.024149] Lustre: Skipped 5 previous similar messages [ 9794.032622] Lustre: Skipped 2 previous similar messages [ 9808.351147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9808.685915] Lustre: server umount lustre-OST0000 complete [ 9818.415168] Lustre: server umount lustre-MDT0000 complete [ 9819.617518] LustreError: 234554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9819.629384] LustreError: 234554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9821.819748] LustreError: 233824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235645 with bad export cookie 9369786929976729326 [ 9821.826380] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9821.833337] LustreError: 233824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9822.390517] Lustre: server umount lustre-MDT0001 complete [ 9918.856244] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [ 9921.038853] Key type lgssc unregistered [ 9921.284288] LNet: 237096:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9921.292661] LNetError: 237096:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9921.314608] LNet: Removed LNI 192.168.202.123@tcp [ 9922.335411] Key type .llcrypt unregistered [ 9922.337104] Key type ._llcrypt unregistered [ 9953.232743] Key type ._llcrypt registered [ 9953.238621] Key type .llcrypt registered [ 9953.378507] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [ 9970.320614] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [ 9971.472889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9971.662478] alg: No test for adler32 (adler32-zlib) [ 9972.830340] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [ 9973.034563] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [ 9974.735490] Key type lgssc registered [ 9975.919543] Lustre: Echo OBD driver; http://www.lustre.org/ [10023.641310] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10034.312129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10034.341525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10035.768905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10035.815783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10035.886453] Lustre: lustre-MDT0000: new disk, initializing [10036.012056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10036.031461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10040.903154] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10051.261292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10051.406689] Lustre: 241121:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10051.449143] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10051.455517] Lustre: Skipped 1 previous similar message [10051.526796] Lustre: lustre-MDT0001: new disk, initializing [10051.620771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10051.649441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10051.662196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10056.299746] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10060.966255] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10066.553164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10066.805162] Lustre: lustre-OST0000: new disk, initializing [10066.809293] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10066.816621] Lustre: 242720:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10066.931079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10072.204335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10072.213765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10072.471688] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10074.301611] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10085.443452] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10085.655452] Lustre: lustre-OST0001: new disk, initializing [10085.658748] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10085.663232] Lustre: 243571:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10085.762828] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10091.651695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10091.672801] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10091.842519] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10093.888780] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10106.856416] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10115.336324] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10127.840086] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10127.843804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10127.858213] Lustre: Skipped 3 previous similar messages [10127.864057] Lustre: Skipped 3 previous similar messages [10132.960886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10132.972868] Lustre: Skipped 3 previous similar messages [10134.176936] Lustre: server umount lustre-MDT0000 complete [10140.471123] LustreError: 241113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781235964 with bad export cookie 15427276954679910865 [10140.485058] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10140.502301] LustreError: 241113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10141.410624] Lustre: server umount lustre-MDT0001 complete [10157.151802] Lustre: server umount lustre-OST0000 complete [10158.304643] Lustre: 238646:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781235966/real 1781235966] req@ffffa0ad2ae76940 x1867761106106496/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781235982 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10158.364270] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10162.146982] Lustre: 238644:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781235969/real 1781235969] req@ffffa0ac100b21c0 x1867761106106752/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781235985 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10162.863913] Lustre: server umount lustre-OST0001 complete [10178.745531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10189.110266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10198.189980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10207.659165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [10228.305119] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10239.608974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10239.635613] Lustre: lustre-MDT0000: reset Object Index mappings [10240.085927] LustreError: 248551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10240.106278] LustreError: 248551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10240.199927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10244.185626] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10245.611451] LustreError: 248552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10249.702618] LustreError: 248551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10252.682591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10252.697787] Lustre: lustre-MDT0001: reset Object Index mappings [10253.203660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10257.482788] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10260.727354] Lustre: 249658:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10268.103987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10274.656787] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10277.800125] LustreError: 250012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10282.983338] LustreError: 250014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10285.079787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10286.290515] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10286.655453] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10286.660527] Lustre: Skipped 1 previous similar message [10291.715759] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [10292.885736] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10300.419317] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10303.796363] Lustre: 251498:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10309.600577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10309.612972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10309.624786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10311.384094] Lustre: server umount lustre-MDT0000 complete [10312.175757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10312.179046] LustreError: 248551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10312.199541] Lustre: Skipped 1 previous similar message [10312.244373] LustreError: 248551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10315.429156] LustreError: 248531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236139 with bad export cookie 15427276954679917655 [10315.437537] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10315.440338] LustreError: 248531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10315.871347] Lustre: server umount lustre-MDT0001 complete [10329.559304] Lustre: server umount lustre-OST0000 complete [10333.472409] Lustre: 238643:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781236141/real 1781236141] req@ffffa0ac0329e1c0 x1867761106156416/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781236157 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10333.507470] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10333.527681] Lustre: Skipped 1 previous similar message [10333.778398] Lustre: server umount lustre-OST0001 complete [10356.192262] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 23:49:37 (1781236177) [10374.852992] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [10388.581930] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10430.245334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10430.578765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10430.605419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10430.656751] Lustre: lustre-MDT0000: new disk, initializing [10430.749639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10430.770248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10435.874514] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10447.176120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10447.377784] Lustre: 255718:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10447.451314] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10447.457309] Lustre: Skipped 1 previous similar message [10447.547516] Lustre: lustre-MDT0001: new disk, initializing [10447.723467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10447.735034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10452.396466] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10461.156805] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10466.885657] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10474.604229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10474.919543] Lustre: lustre-OST0000: new disk, initializing [10474.929437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10474.939489] Lustre: 257462:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10476.587854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10476.591060] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10476.743120] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10482.696673] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10491.142209] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10495.999624] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10496.432772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10502.024634] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10502.334819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10507.232372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10507.242852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10507.285150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10511.221551] Lustre: server umount lustre-OST0000 complete [10517.990328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10518.011836] Lustre: Skipped 2 previous similar messages [10528.226093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10528.249937] Lustre: Skipped 3 previous similar messages [10531.295277] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10531.672360] Lustre: server umount lustre-MDT0000 complete [10533.367786] LustreError: 255726:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10533.398649] LustreError: 255726:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10536.335062] LustreError: 255711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236360 with bad export cookie 15427276954679919748 [10536.346668] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10536.354955] LustreError: 255711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10536.693328] Lustre: server umount lustre-MDT0001 complete [10561.738620] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10573.014603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10573.502341] LustreError: 260257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10573.569496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10573.577097] Lustre: Skipped 2 previous similar messages [10579.701649] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10589.775737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10594.914806] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10598.237642] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10605.645316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10613.003696] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10621.594366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10621.857437] Lustre: lustre-OST0001: new disk, initializing [10621.860671] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10621.875303] Lustre: 262512:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10623.875135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10623.879662] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10623.996708] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10628.580725] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10639.758681] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10643.638256] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10649.904678] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10649.941813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10656.737181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10656.759636] LustreError: Skipped 1 previous similar message [10656.764164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10656.784731] Lustre: Skipped 3 previous similar messages [10656.789069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10663.212568] Lustre: server umount lustre-MDT0000 complete [10664.936979] LustreError: 262323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10664.971634] LustreError: 262323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [10667.338553] LustreError: 260237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236491 with bad export cookie 15427276954679920630 [10667.350124] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10667.360507] LustreError: 260237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10667.947669] Lustre: server umount lustre-MDT0001 complete [10683.094053] Lustre: server umount lustre-OST0000 complete [10686.241578] Lustre: 238643:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781236493/real 1781236493] req@ffffa0ad3dc68000 x1867761106272256/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781236509 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10687.371852] Lustre: server umount lustre-OST0001 complete [10705.855401] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10716.540757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10716.985991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10716.995283] Lustre: Skipped 3 previous similar messages [10721.734953] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10731.307643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10731.488656] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10731.501763] Lustre: Skipped 4 previous similar messages [10737.211996] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10740.187761] Lustre: 266646:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10747.516338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10754.179729] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10762.382865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10763.646931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10769.454894] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10771.149621] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10780.342441] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10784.683197] Lustre: 268484:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10790.925279] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10790.973499] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10802.295372] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 23:57:04 (1781236624) [10818.529675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10818.546841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10818.572761] Lustre: Skipped 4 previous similar messages [10818.588698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10823.614255] Lustre: server umount lustre-MDT0000 complete [10825.718462] LustreError: 266485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10825.764650] LustreError: 266485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [10827.712467] LustreError: 266262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236651 with bad export cookie 15427276954679927420 [10827.733202] LustreError: 266262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10827.746363] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10842.079130] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10842.333639] Lustre: server umount lustre-MDT0001 complete [10853.944260] Lustre: server umount lustre-OST0000 complete [10858.633767] Lustre: server umount lustre-OST0001 complete [10865.666701] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [10876.322335] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10920.648360] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [10929.916470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10930.159782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10930.197959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10930.266641] Lustre: lustre-MDT0000: new disk, initializing [10930.351623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10934.765436] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10946.651770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10946.791173] Lustre: 272673:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10947.051336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10951.832463] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10964.115821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10964.500503] Lustre: 274272:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10966.232958] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10970.488982] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10980.609076] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10980.774244] Lustre: 275127:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10980.850257] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10980.859759] Lustre: Skipped 6 previous similar messages [10981.930475] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10981.947033] Lustre: Skipped 1 previous similar message [10981.993010] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10987.481609] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [10997.928228] Lustre: DEBUG MARKER: Using TIMEOUT=20 [11001.298651] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [11001.303376] Lustre: Skipped 1 previous similar message [11015.821483] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 00:00:37 (1781236837) [11017.711190] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11017.719105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11017.729422] Lustre: Skipped 6 previous similar messages [11017.741352] Lustre: Skipped 13 previous similar messages [11024.072858] Lustre: server umount lustre-MDT0000 complete [11027.989651] LustreError: 272663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781236851 with bad export cookie 15427276954679934847 [11027.999306] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11028.009024] LustreError: 272663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11028.303955] Lustre: server umount lustre-MDT0001 complete [11041.441904] Lustre: server umount lustre-OST0000 complete [11055.841547] Lustre: server umount lustre-OST0001 complete [11065.438365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11070.546911] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11078.785334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11083.818308] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11089.552424] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11094.220183] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11101.604989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11104.114128] LustreError: 278971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11104.137174] LustreError: 278971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [11108.373323] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11113.957879] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11132.346140] loop1: detected capacity change from 0 to 204800000 [11132.775648] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11132.810853] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11132.850370] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11139.180635] loop1: detected capacity change from 0 to 204800000 [11144.859634] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11144.961678] LustreError: 277359:0:(mgs_llog.c:673:mgs_find_or_make_fsdb_nolock()) fs13728: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [11144.971206] LustreError: 277359:0:(mgs_llog.c:731:mgs_check_index()) Can't get db for fs13728 [11144.976142] LustreError: 277359:0:(mgs_handler.c:517:mgs_target_reg()) fs13728-MDT0001: Error checking target index: rc = -13 [11144.984854] LustreError: fs13728-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11144.991808] LustreError: 280746:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [11144.996708] LustreError: 280746:0:(tgt_mount.c:2084:server_put_super()) no obd fs13728-MDT0001 [11145.000566] LustreError: 280746:0:(tgt_mount.c:116:server_deregister_mount()) fs13728-MDT0001 not registered [11145.023612] Lustre: server umount fs13728-MDT0001 complete [11145.026153] LustreError: 280746:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11154.238380] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11154.302282] Lustre: Setting parameter fs13728-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs13728-MDT0001 [11154.350707] Lustre: srv-fs13728-MDT0001: No data found on store. Initialize space. [11154.354613] Lustre: Skipped 4 previous similar messages [11154.422849] Lustre: fs13728-MDT0001: new disk, initializing [11154.426074] Lustre: Skipped 3 previous similar messages [11154.523847] Lustre: srv-fs13728-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [11154.529462] Lustre: 281213:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13728-MDT0001-mdtlov: can't create new llog: rc = -115 [11154.537803] LustreError: 281213:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13728-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11158.619640] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11165.688985] Lustre: server umount fs13728-MDT0001 complete [11179.099895] loop1: detected capacity change from 0 to 204800000 [11184.499946] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11185.014759] Lustre: 282763:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13728-MDT0001-mdtlov: can't create new llog: rc = -115 [11185.032335] LustreError: 282763:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13728-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11187.175160] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11190.073939] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11197.174209] Lustre: server umount fs13728-MDT0001 complete [11212.265182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11212.280745] LustreError: Skipped 1 previous similar message [11212.282746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11212.300737] Lustre: Skipped 2 previous similar messages [11214.104332] Lustre: server umount lustre-MDT0000 complete [11223.067753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11223.151225] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11228.657478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11228.747088] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11237.746744] loop1: detected capacity change from 0 to 204800000 [11244.876757] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11245.288510] Lustre: 285177:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs13728-MDT0001-mdtlov: can't create new llog: rc = -115 [11245.294030] LustreError: 285177:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs13728-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [11251.154482] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11256.155772] Lustre: server umount fs13728-MDT0001 complete [11278.308860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11278.323188] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11278.331977] Lustre: Skipped 8 previous similar messages [11284.245918] Lustre: server umount lustre-OST0000 complete [11294.459389] Lustre: server umount lustre-MDT0000 complete [11298.027244] LustreError: 277350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237121 with bad export cookie 15427276954679937528 [11298.046929] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11298.054108] LustreError: 277350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11298.653030] Lustre: server umount lustre-MDT0001 complete [11310.037300] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [11312.783545] Key type lgssc unregistered [11313.045672] LNet: 287488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11313.075413] LNetError: 287488:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11314.161410] LNet: Removed LNI 192.168.202.123@tcp [11314.858176] Key type .llcrypt unregistered [11314.863172] Key type ._llcrypt unregistered [11333.146703] Key type ._llcrypt registered [11333.149634] Key type .llcrypt registered [11333.257072] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [11334.277077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11334.288616] alg: No test for adler32 (adler32-zlib) [11335.405353] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [11335.578747] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [11337.231291] Key type lgssc registered [11338.315424] Lustre: Echo OBD driver; http://www.lustre.org/ [11347.477631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11347.492356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11348.969450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11353.392604] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11359.202281] LustreError: 288742:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11359.223125] LustreError: 288742:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11361.699527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11362.373818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11369.422685] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11377.247594] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11381.715731] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11389.289254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11389.823411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11393.963291] LustreError: 290569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11396.241042] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11399.150834] LustreError: 290345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11402.813370] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11404.264072] LustreError: 290346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11409.674528] LustreError: 290344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11409.704190] LustreError: 290344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11419.907262] LustreError: 290345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11419.924843] LustreError: 290345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11427.819100] loop1: detected capacity change from 0 to 204800000 [11428.098732] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11428.123996] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11428.132650] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11435.089625] loop1: detected capacity change from 0 to 204800000 [11437.030505] LustreError: 290569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11437.073880] LustreError: 290569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [11441.142762] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11441.274984] Lustre: fs6668-OST0000: new disk, initializing [11441.278970] Lustre: srv-fs6668-OST0000: No data found on store. Initialize space. [11441.285491] Lustre: 292138:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11441.350926] Lustre: fs6668-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11449.196323] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11459.544634] Lustre: server umount fs6668-OST0000 complete [11471.104519] LustreError: 290345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11471.135093] LustreError: 290345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 18 previous similar messages [11472.894809] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11477.491124] loop1: detected capacity change from 0 to 204800000 [11485.007764] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11485.179417] Lustre: fs6668-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11488.239978] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11491.573679] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11500.280025] loop2: detected capacity change from 0 to 204800000 [11500.591851] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11500.604212] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11500.616413] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11506.837356] loop2: detected capacity change from 0 to 204800000 [11513.792403] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11513.935813] LustreError: fs6668-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11513.955928] LustreError: fs6668-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11513.970625] LustreError: 295146:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -13 [11513.979260] LustreError: 295146:0:(tgt_mount.c:2084:server_put_super()) no obd fs6668-OST0001 [11513.988597] LustreError: 295146:0:(tgt_mount.c:116:server_deregister_mount()) fs6668-OST0001 not registered [11514.061509] Lustre: server umount fs6668-OST0001 complete [11514.075740] LustreError: 295146:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -13 [11525.467388] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11525.729140] Lustre: fs6668-OST0001: new disk, initializing [11525.743881] Lustre: srv-fs6668-OST0001: No data found on store. Initialize space. [11525.765930] Lustre: 295697:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11525.916497] Lustre: fs6668-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11533.537908] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11537.657550] LustreError: 291319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11537.696957] LustreError: 291319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 38 previous similar messages [11544.101174] Lustre: server umount fs6668-OST0000 complete [11556.386702] Lustre: server umount fs6668-OST0001 complete [11570.930233] loop1: detected capacity change from 0 to 204800000 [11577.104079] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11577.354168] Lustre: fs6668-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11583.680825] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11591.915842] loop2: detected capacity change from 0 to 204800000 [11597.858157] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11605.793558] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11615.715675] Lustre: server umount fs6668-OST0000 complete [11625.188729] Lustre: server umount fs6668-OST0001 complete [11626.475260] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11639.776726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11639.796135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11639.827975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11641.831442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11641.843520] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11643.409899] Lustre: server umount lustre-OST0000 complete [11648.992194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11649.001754] LustreError: Skipped 2 previous similar messages [11649.007574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11649.020484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11649.025856] Lustre: Skipped 1 previous similar message [11652.065030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11652.066956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11653.812664] Lustre: server umount lustre-MDT0000 complete [11657.934968] LustreError: 295696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237481 with bad export cookie 10178752523137442113 [11657.937337] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11657.946452] LustreError: 295696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11658.308951] Lustre: server umount lustre-MDT0001 complete [11668.645598] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [11671.651744] Key type lgssc unregistered [11672.097307] LNet: 301556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11672.111417] LNetError: 301556:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11672.136887] LNet: Removed LNI 192.168.202.123@tcp [11673.300187] Key type .llcrypt unregistered [11673.303954] Key type ._llcrypt unregistered [11708.132876] Key type ._llcrypt registered [11708.138793] Key type .llcrypt registered [11708.262536] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [11725.287422] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [11726.542805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11726.582610] alg: No test for adler32 (adler32-zlib) [11727.812765] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [11728.087055] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [11729.783191] Key type lgssc registered [11730.899222] Lustre: Echo OBD driver; http://www.lustre.org/ [11772.364727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11772.399541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11773.816073] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11773.929788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11774.104790] Lustre: lustre-MDT0000: new disk, initializing [11774.306599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11774.377902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11778.808962] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11792.588312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11792.692381] Lustre: 305382:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11792.757807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11792.760858] Lustre: Skipped 1 previous similar message [11792.803262] Lustre: lustre-MDT0001: new disk, initializing [11792.861524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11792.902368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11792.911164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11798.232368] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11806.568586] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11811.304934] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11820.055813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11820.377330] Lustre: lustre-OST0000: new disk, initializing [11820.380653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11820.385884] Lustre: 307126:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11820.495157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11826.750328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11826.776519] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11826.934556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11827.731391] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [11836.139850] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11838.203255] Lustre: Mounted lustre-client [11839.108212] Lustre: Unmounted lustre-client [11847.649644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11847.655438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11847.660257] LustreError: Skipped 1 previous similar message [11847.662103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11847.685753] Lustre: Skipped 1 previous similar message [11848.677575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11848.683090] Lustre: Skipped 2 previous similar messages [11854.055661] Lustre: server umount lustre-OST0000 complete [11864.034633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11864.064425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11866.125232] Lustre: server umount lustre-MDT0000 complete [11869.156910] LustreError: 306574:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11869.183749] LustreError: 306574:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11870.633620] LustreError: 307125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237694 with bad export cookie 1602284367211660940 [11870.644896] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11870.657812] LustreError: 307125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11871.365919] Lustre: server umount lustre-MDT0001 complete [11883.622355] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [11886.543860] Key type lgssc unregistered [11897.120419] LNet: 309163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11897.131024] LNetError: 309163:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11969.896968] LNet: Removed LNI 192.168.202.123@tcp [11971.114954] Key type .llcrypt unregistered [11971.119219] Key type ._llcrypt unregistered [11990.670995] Key type ._llcrypt registered [11990.677523] Key type .llcrypt registered [11990.884993] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [12006.810848] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [12007.937028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12008.115352] alg: No test for adler32 (adler32-zlib) [12009.374418] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [12009.631679] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [12011.335570] Key type lgssc registered [12012.435324] Lustre: Echo OBD driver; http://www.lustre.org/ [12045.475096] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 00:17:47 (1781237867) [12053.047080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12053.078117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12054.535642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12054.639205] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12054.831352] Lustre: lustre-MDT0000: new disk, initializing [12054.970732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12054.993948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12060.230400] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12071.189708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12071.327265] Lustre: 312350:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12071.391199] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12071.397326] Lustre: Skipped 1 previous similar message [12071.489535] Lustre: lustre-MDT0001: new disk, initializing [12071.639242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12071.701962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12071.714971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12076.121125] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12083.438741] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12087.235399] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12094.253937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12094.594924] Lustre: lustre-OST0000: new disk, initializing [12094.601270] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12094.606300] Lustre: 314092:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12094.725594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12102.405220] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12104.307943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12104.322834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12104.543862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12112.103363] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12125.158057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12125.174045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12125.184816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12125.192864] Lustre: Skipped 1 previous similar message [12125.220631] Lustre: Skipped 1 previous similar message [12129.373742] Lustre: server umount lustre-OST0000 complete [12135.395186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12135.424729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12135.430588] Lustre: Skipped 1 previous similar message [12139.867731] Lustre: server umount lustre-MDT0000 complete [12144.560427] LustreError: 312341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781237968 with bad export cookie 14877203176262052673 [12144.572907] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12144.586245] LustreError: 312341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12145.000251] Lustre: server umount lustre-MDT0001 complete [12158.771950] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [12162.083598] Key type lgssc unregistered [12162.456715] LNet: 315725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12162.461570] LNetError: 315725:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12163.495664] LNet: Removed LNI 192.168.202.123@tcp [12164.477361] Key type .llcrypt unregistered [12164.479250] Key type ._llcrypt unregistered [12192.272485] Key type ._llcrypt registered [12192.276203] Key type .llcrypt registered [12192.396359] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [12193.918583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12194.232067] alg: No test for adler32 (adler32-zlib) [12195.544508] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [12195.890252] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [12197.567346] Key type lgssc registered [12199.117146] Lustre: Echo OBD driver; http://www.lustre.org/ [12208.728112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12208.761581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12210.597931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12215.214791] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12220.924521] LustreError: 317132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12220.946130] LustreError: 317132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12225.032696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12225.437789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12229.951978] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12235.364872] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12239.119693] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12245.464751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12245.823782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12251.841634] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12256.305391] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12263.080293] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12264.748794] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12271.593058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12271.595587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12271.596387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12271.616092] Lustre: Skipped 2 previous similar messages [12273.434578] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.23@tcp (stopping) [12273.436622] Lustre: Skipped 2 previous similar messages [12274.853032] Lustre: server umount lustre-MDT0000 complete [12276.718799] LustreError: 319244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12276.740354] LustreError: 319244:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12278.207927] LustreError: 317112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238101 with bad export cookie 13226868654154322716 [12278.213169] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12278.215570] LustreError: 317112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12278.511739] LustreError: 317126:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.23@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12278.537065] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.23@tcp (stopping) [12281.827048] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12281.837365] LustreError: 317127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12281.840135] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12284.644456] Lustre: server umount lustre-MDT0001 complete [12289.199220] Lustre: server umount lustre-OST0000 complete [12298.463190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12302.575927] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12305.295187] Lustre: 320858:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.123@tcp [12307.959534] Lustre: server umount MGS complete [12317.280836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12317.786212] LustreError: 321430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12317.810203] LustreError: 321430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12317.873933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12322.135317] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12327.396494] LustreError: 321431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12327.430196] LustreError: 321431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12331.314623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12331.755603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12337.033691] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12345.580650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12345.929434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12351.280752] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12355.967766] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12357.306304] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12361.052063] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12367.345589] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [12373.473774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12373.485716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12373.513484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12373.517348] Lustre: Skipped 2 previous similar messages [12379.016225] Lustre: server umount lustre-OST0000 complete [12383.712729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12383.722250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12383.732571] Lustre: Skipped 2 previous similar messages [12383.743595] Lustre: Skipped 4 previous similar messages [12389.638044] Lustre: server umount lustre-MDT0000 complete [12393.170074] LustreError: 321411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238216 with bad export cookie 13226868654154323556 [12393.179851] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12393.185913] LustreError: 321411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12393.471943] Lustre: server umount lustre-MDT0001 complete [12406.786402] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [12409.851296] Key type lgssc unregistered [12410.149100] LNet: 324604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12410.158726] LNetError: 324604:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12410.180603] LNet: Removed LNI 192.168.202.123@tcp [12411.044952] Key type .llcrypt unregistered [12411.046939] Key type ._llcrypt unregistered [12429.691236] Key type ._llcrypt registered [12429.694989] Key type .llcrypt registered [12429.822438] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [12445.645541] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [12446.767189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12446.827244] alg: No test for adler32 (adler32-zlib) [12447.910972] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [12448.172424] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [12449.823411] Key type lgssc registered [12451.233949] Lustre: Echo OBD driver; http://www.lustre.org/ [12490.983596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12491.026661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12492.273882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12492.300271] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12492.401638] Lustre: lustre-MDT0000: new disk, initializing [12492.503273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12492.553880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12497.492796] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12508.717346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12508.857489] Lustre: 327730:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12508.935902] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12508.946959] Lustre: Skipped 1 previous similar message [12509.086365] Lustre: lustre-MDT0001: new disk, initializing [12509.230651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12509.319208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12509.339772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12515.095852] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12523.501330] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12526.983821] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12534.168721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12534.420618] Lustre: lustre-OST0000: new disk, initializing [12534.426529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12534.432141] Lustre: 329474:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12534.534640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12540.970854] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12542.063601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12542.077389] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12542.283882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12548.591685] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12555.225911] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12555.496790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12560.497604] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12560.781704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12568.033907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12568.039341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12568.053093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12568.081273] Lustre: Skipped 1 previous similar message [12569.407874] Lustre: server umount lustre-OST0000 complete [12573.154428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12573.160982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12573.166821] Lustre: Skipped 2 previous similar messages [12573.178695] Lustre: Skipped 1 previous similar message [12578.273173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12578.280784] Lustre: Skipped 1 previous similar message [12579.418028] Lustre: server umount lustre-MDT0000 complete [12583.277643] LustreError: 327721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238407 with bad export cookie 16631917613058083591 [12583.280302] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12583.291967] LustreError: 327721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12583.938837] Lustre: server umount lustre-MDT0001 complete [12594.279488] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 00:26:56 (1781238416) [12613.950937] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [12624.990374] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [12666.780839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12667.292772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12667.389067] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12667.494305] Lustre: lustre-MDT0000: new disk, initializing [12667.617900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12667.664437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12674.627296] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12687.952199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12688.140104] Lustre: 334057:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12688.229299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12688.234200] Lustre: Skipped 1 previous similar message [12688.321260] Lustre: lustre-MDT0001: new disk, initializing [12688.392239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12688.435963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12688.458690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12693.563181] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12701.496335] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12705.860954] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12713.183825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12713.459486] Lustre: lustre-OST0000: new disk, initializing [12713.463153] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12713.472971] Lustre: 335798:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12713.596447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12719.160982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12719.169506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12719.291807] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12719.947697] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [12727.805769] Lustre: DEBUG MARKER: oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12755.424610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12755.433393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12755.443023] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12755.447542] Lustre: Skipped 1 previous similar message [12757.604267] Lustre: server umount lustre-OST0000 complete [12765.155980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12765.168612] LustreError: Skipped 1 previous similar message [12765.174852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12765.191063] Lustre: Skipped 1 previous similar message [12765.198716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12768.259533] Lustre: server umount lustre-MDT0000 complete [12771.711964] LustreError: 334049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781238595 with bad export cookie 16631917613058084473 [12771.713755] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12771.729030] LustreError: 334049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12772.140235] Lustre: server umount lustre-MDT0001 complete [12782.192388] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [12785.609341] Key type lgssc unregistered [12785.889320] LNet: 337819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12785.895920] LNetError: 337819:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12785.921879] LNet: Removed LNI 192.168.202.123@tcp [12786.746838] Key type .llcrypt unregistered [12786.750063] Key type ._llcrypt unregistered [12880.173193] Key type ._llcrypt registered [12880.175410] Key type .llcrypt registered [12880.345171] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [12880.731431] Key type .llcrypt unregistered [12880.733868] Key type ._llcrypt unregistered [12969.651888] Key type ._llcrypt registered [12969.661837] Key type .llcrypt registered [12969.850335] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [12970.238484] Key type .llcrypt unregistered [12970.243137] Key type ._llcrypt unregistered [13000.995757] Key type ._llcrypt registered [13001.000762] Key type .llcrypt registered [13001.143956] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13001.454856] Key type .llcrypt unregistered [13001.460227] Key type ._llcrypt unregistered [13035.372187] Key type ._llcrypt registered [13035.373475] Key type .llcrypt registered [13035.484889] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13035.768380] Key type .llcrypt unregistered [13035.770232] Key type ._llcrypt unregistered [13123.393156] Key type ._llcrypt registered [13123.394739] Key type .llcrypt registered [13123.600791] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13123.893490] Key type .llcrypt unregistered [13123.895374] Key type ._llcrypt unregistered [13156.922255] Key type ._llcrypt registered [13156.926899] Key type .llcrypt registered [13157.122934] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13157.420393] Key type .llcrypt unregistered [13157.424849] Key type ._llcrypt unregistered [13192.945086] Key type ._llcrypt registered [13192.949580] Key type .llcrypt registered [13193.265607] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13193.643345] Key type .llcrypt unregistered [13193.645058] Key type ._llcrypt unregistered [13227.458602] Key type ._llcrypt registered [13227.462962] Key type .llcrypt registered [13227.649400] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13227.997502] Key type .llcrypt unregistered [13228.001633] Key type ._llcrypt unregistered [13263.658847] Key type ._llcrypt registered [13263.667480] Key type .llcrypt registered [13263.852706] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13264.374794] Key type .llcrypt unregistered [13264.382943] Key type ._llcrypt unregistered [13302.698278] Key type ._llcrypt registered [13302.701098] Key type .llcrypt registered [13302.776090] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13303.186829] Key type .llcrypt unregistered [13303.191657] Key type ._llcrypt unregistered [13369.110611] Key type ._llcrypt registered [13369.113638] Key type .llcrypt registered [13369.322623] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13369.516304] Key type .llcrypt unregistered [13369.518761] Key type ._llcrypt unregistered [13405.370294] Key type ._llcrypt registered [13405.372105] Key type .llcrypt registered [13405.568341] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13405.955341] Key type .llcrypt unregistered [13405.960148] Key type ._llcrypt unregistered [13443.682929] Key type ._llcrypt registered [13443.698317] Key type .llcrypt registered [13443.877129] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13444.448499] Key type .llcrypt unregistered [13444.451175] Key type ._llcrypt unregistered [13505.713529] Key type ._llcrypt registered [13505.715913] Key type .llcrypt registered [13505.785722] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [13522.286057] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [13523.368083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13523.546067] alg: No test for adler32 (adler32-zlib) [13524.598718] Lustre: Lustre: Build Version: 2.17.53_68_ga4ebd63 [13524.852757] LNet: Added LNI 192.168.202.123@tcp [8/256/0/180] [13526.535314] Key type lgssc registered [13527.637907] Lustre: Echo OBD driver; http://www.lustre.org/ [13565.074446] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13565.117124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13566.509444] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13566.559573] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13566.674075] Lustre: tst250fs-MDT0000: new disk, initializing [13566.783632] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13566.809443] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13571.455786] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13582.367647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13582.472720] Lustre: 357925:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13582.515226] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13582.520788] Lustre: Skipped 1 previous similar message [13582.611148] Lustre: tst250fs-MDT0001: new disk, initializing [13582.715423] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13582.777587] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13582.789623] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13588.344117] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13594.280664] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13601.934678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13602.402884] Lustre: tst250fs-OST0000: new disk, initializing [13602.409977] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13602.426569] Lustre: 359528:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13602.623483] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13603.423485] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13603.437777] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13603.610774] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13609.430138] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13622.560146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13622.921482] Lustre: tst250fs-OST0001: new disk, initializing [13622.925923] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13622.942793] Lustre: 360382:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13623.091280] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13629.127803] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13629.143663] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13629.316579] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13631.232446] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_default_debug -1 all [13637.790951] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13640.131474] Lustre: Setting parameter tst250fs.llite.tst250fs-*.max_read_ahead_mb=32 in log params [13640.138613] Lustre: Skipped 1 previous similar message [13660.133418] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13660.145499] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13660.160405] Lustre: Skipped 2 previous similar messages [13660.182521] Lustre: Skipped 3 previous similar messages [13665.253259] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13665.258782] Lustre: Skipped 2 previous similar messages [13666.417887] Lustre: server umount tst250fs-MDT0000 complete [13671.343135] LustreError: 360979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781239495 with bad export cookie 16315634064740153519 [13671.349184] LustreError: MGC192.168.202.123@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13671.367164] LustreError: 360979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13672.339946] Lustre: server umount tst250fs-MDT0001 complete [13687.191868] Lustre: server umount tst250fs-OST0000 complete [13701.225806] Lustre: server umount tst250fs-OST0001 complete [13718.940500] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [13725.990425] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [13756.522021] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 00:46:18 (1781239578) [13758.114574] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13777.379323] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing set_hostid [13793.601139] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing load_modules_local [13828.525849] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13447 sec ========= 00:47:30 (1781239650) [13831.286655] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:47:32 (1781239652) === [13835.877651] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:47:37 (1781239657) === [13879.249751] Lustre: DEBUG MARKER: oleg223-server.virtnet: executing unload_modules_local [13882.802674] Key type lgssc unregistered [13883.056259] LNet: 369439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13883.065985] LNetError: 369439:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13883.081485] LNet: Removed LNI 192.168.202.123@tcp [13883.876469] Key type .llcrypt unregistered [13883.878944] Key type ._llcrypt unregistered