[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 483672245 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002364] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.006917] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009155] LSM: Security Framework initializing [ 0.010055] Yama: becoming mindful. [ 0.011048] SELinux: Initializing. [ 0.012044] *** VALIDATE selinux *** [ 0.020676] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024715] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027109] *** VALIDATE tmpfs *** [ 0.029225] *** VALIDATE proc *** [ 0.030279] *** VALIDATE cgroup *** [ 0.031012] *** VALIDATE cgroup2 *** [ 0.032273] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035034] Spectre V2 : User space: Vulnerable [ 0.036011] Speculative Store Bypass: Vulnerable [ 0.039345] debug: unmapping init [mem 0xffffffffa6459000-0xffffffffa6460fff] [ 0.041197] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042667] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043025] ... version: 2 [ 0.044014] ... bit width: 48 [ 0.045015] ... generic registers: 4 [ 0.046016] ... value mask: 0000ffffffffffff [ 0.047016] ... max period: 00007fffffffffff [ 0.048016] ... fixed-purpose events: 3 [ 0.049016] ... event mask: 000000070000000f [ 0.050295] rcu: Hierarchical SRCU implementation. [ 0.052420] smp: Bringing up secondary CPUs ... [ 0.053599] x86: Booting SMP configuration: [ 0.054029] .... node #0, CPUs: #1 #2 #3 [ 0.057320] smp: Brought up 1 node, 4 CPUs [ 0.059014] smpboot: Max logical packages: 1 [ 0.060020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.097297] node 0 deferred pages initialised in 35ms [ 0.100112] devtmpfs: initialized [ 0.102105] x86/mm: Memory block size: 128MB [ 0.104803] gcov: version magic: 0x41383552 [ 0.106384] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.110105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.113290] pinctrl core: initialized pinctrl subsystem [ 0.115191] [ 0.115887] ************************************************************* [ 0.119013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.122012] ** ** [ 0.124017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.127013] ** ** [ 0.130014] ** This means that this kernel is built to expose internal ** [ 0.133013] ** IOMMU data structures, which may compromise security on ** [ 0.136016] ** your system. ** [ 0.139013] ** ** [ 0.141014] ** If you see this message and you are not debugging the ** [ 0.144012] ** kernel, report this immediately to your vendor! ** [ 0.146011] ** ** [ 0.149013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152013] ************************************************************* [ 0.154765] NET: Registered protocol family 16 [ 0.157447] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.160054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.163064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.167085] cpuidle: using governor menu [ 0.168646] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.171464] PCI: Using configuration type 1 for base access [ 0.174181] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.183371] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.184019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.186073] cryptd: max_cpu_qlen set to 1000 [ 0.189042] ACPI: Added _OSI(Module Device) [ 0.190010] ACPI: Added _OSI(Processor Device) [ 0.192010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.193014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.198388] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.205493] ACPI: Interpreter enabled [ 0.207069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.208016] ACPI: Using IOAPIC for interrupt routing [ 0.210116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.214396] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.226165] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.229061] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.232022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.236077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.241504] acpiphp: Slot [2] registered [ 0.243127] acpiphp: Slot [5] registered [ 0.245108] acpiphp: Slot [6] registered [ 0.247131] acpiphp: Slot [7] registered [ 0.248116] acpiphp: Slot [8] registered [ 0.250119] acpiphp: Slot [9] registered [ 0.252116] acpiphp: Slot [10] registered [ 0.254131] acpiphp: Slot [3] registered [ 0.255172] acpiphp: Slot [4] registered [ 0.257145] acpiphp: Slot [11] registered [ 0.259108] acpiphp: Slot [12] registered [ 0.260085] acpiphp: Slot [13] registered [ 0.261080] acpiphp: Slot [14] registered [ 0.262093] acpiphp: Slot [15] registered [ 0.264101] acpiphp: Slot [16] registered [ 0.266096] acpiphp: Slot [17] registered [ 0.267134] acpiphp: Slot [18] registered [ 0.269108] acpiphp: Slot [19] registered [ 0.271125] acpiphp: Slot [20] registered [ 0.273097] acpiphp: Slot [21] registered [ 0.274116] acpiphp: Slot [22] registered [ 0.276124] acpiphp: Slot [23] registered [ 0.278123] acpiphp: Slot [24] registered [ 0.279085] acpiphp: Slot [25] registered [ 0.281101] acpiphp: Slot [26] registered [ 0.282097] acpiphp: Slot [27] registered [ 0.284104] acpiphp: Slot [28] registered [ 0.285104] acpiphp: Slot [29] registered [ 0.287105] acpiphp: Slot [30] registered [ 0.289109] acpiphp: Slot [31] registered [ 0.290065] PCI host bridge to bus 0000:00 [ 0.292021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.295024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.297021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.300024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.302020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.305027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.306167] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.309611] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.313189] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.323015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.328060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.331019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.333051] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.335018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.338783] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.343117] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.345042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.348883] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.354013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.368017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.373019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.378997] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.389025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.395023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.412035] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.422820] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.432016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.439016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.457024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.467802] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.480021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.486019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.518015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.529141] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.536029] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.542024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.554033] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.569747] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.575021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.581021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.596023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.605830] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.614015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.625016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.642017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.654402] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.657478] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.660447] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.662374] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.666320] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.673150] iommu: Default domain type: Passthrough [ 0.674517] SCSI subsystem initialized [ 0.676202] ACPI: bus type USB registered [ 0.677152] usbcore: registered new interface driver usbfs [ 0.680121] usbcore: registered new interface driver hub [ 0.682137] usbcore: registered new device driver usb [ 0.684362] pps_core: LinuxPPS API ver. 1 registered [ 0.686018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.689086] PTP clock support registered [ 0.692055] EDAC MC: Ver: 3.0.0 [ 0.693611] PCI: Using ACPI for IRQ routing [ 0.695777] NetLabel: Initializing [ 0.696008] NetLabel: domain hash size = 128 [ 0.698009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.699070] NetLabel: unlabeled traffic allowed by default [ 0.701130] vgaarb: loaded [ 0.702312] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.705024] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.710327] clocksource: Switched to clocksource kvm-clock [ 0.816521] VFS: Disk quotas dquot_6.6.0 [ 0.818790] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.821843] *** VALIDATE ramfs *** [ 0.823467] *** VALIDATE hugetlbfs *** [ 0.825519] pnp: PnP ACPI init [ 0.827930] pnp: PnP ACPI: found 6 devices [ 0.844144] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.847545] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.849811] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.852087] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.854587] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.856987] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.859446] NET: Registered protocol family 2 [ 0.861568] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.865593] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.868986] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.874898] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.878379] TCP: Hash tables configured (established 65536 bind 65536) [ 0.881092] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.884458] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.887641] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.890864] NET: Registered protocol family 1 [ 0.893642] RPC: Registered named UNIX socket transport module. [ 0.895588] RPC: Registered udp transport module. [ 0.897388] RPC: Registered tcp transport module. [ 0.898943] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.901277] NET: Registered protocol family 44 [ 0.902608] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.904510] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.906336] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.908442] PCI: CLS 0 bytes, default 64 [ 0.909871] Unpacking initramfs... [ 2.293163] debug: unmapping init [mem 0xffff9f027cc54000-0xffff9f027ffbffff] [ 2.298801] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.301434] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.304960] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.770105] Initialise system trusted keyrings [ 2.771781] Key type blacklist registered [ 2.773849] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.782470] zbud: loaded [ 2.785122] *** VALIDATE nfs *** [ 2.786250] *** VALIDATE nfs4 *** [ 2.787475] pstore: using deflate compression [ 2.790444] Platform Keyring initialized [ 2.898449] NET: Registered protocol family 38 [ 2.903100] Key type asymmetric registered [ 2.904843] Asymmetric key parser 'x509' registered [ 2.907095] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.910932] io scheduler mq-deadline registered [ 2.913066] io scheduler kyber registered [ 2.914937] io scheduler bfq registered [ 2.917063] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.920634] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.924041] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.927628] ACPI: Power Button [PWRF] [ 2.933428] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.940366] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.959355] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.967421] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.985307] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.014768] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.048323] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.054139] Non-volatile memory driver v1.3 [ 3.055970] Linux agpgart interface v0.103 [ 3.089078] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.092618] vda: detected capacity change from 0 to 74702848 [ 3.107863] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.111295] vdb: detected capacity change from 0 to 1073741824 [ 3.127504] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.131133] vdc: detected capacity change from 0 to 2621440000 [ 3.146978] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.150347] vdd: detected capacity change from 0 to 2621440000 [ 3.167643] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.171322] vde: detected capacity change from 0 to 4294967296 [ 3.185568] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.188947] vdf: detected capacity change from 0 to 4294967296 [ 3.204446] libphy: Fixed MDIO Bus: probed [ 3.213694] usbcore: registered new interface driver usbserial_generic [ 3.216491] usbserial: USB Serial support registered for generic [ 3.219447] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.224258] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.226321] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.229107] mousedev: PS/2 mouse device common for all mice [ 3.232214] rtc_cmos 00:05: RTC can wake from S4 [ 3.235509] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.239614] rtc_cmos 00:05: registered as rtc0 [ 3.241990] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.242376] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.248219] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.249350] intel_pstate: CPU model not supported [ 3.256227] hid: raw HID events driver (C) Jiri Kosina [ 3.258934] usbcore: registered new interface driver usbhid [ 3.261485] usbhid: USB HID core driver [ 3.263266] drop_monitor: Initializing network drop monitor service [ 3.266176] Initializing XFRM netlink socket [ 3.268943] NET: Registered protocol family 10 [ 3.272186] Segment Routing with IPv6 [ 3.273989] NET: Registered protocol family 17 [ 3.276040] mpls_gso: MPLS GSO support [ 3.281495] RAS: Correctable Errors collector initialized. [ 3.284122] AVX version of gcm_enc/dec engaged. [ 3.286085] AES CTR mode by8 optimization enabled [ 3.375320] sched_clock: Marking stable (3375252520, 0)->(4311009448, -935756928) [ 3.378232] registered taskstats version 1 [ 3.381124] Loading compiled-in X.509 certificates [ 3.382609] zswap: loaded using pool lzo/zbud [ 3.404186] Key type big_key registered [ 3.417188] Key type encrypted registered [ 3.419221] ima: No TPM chip found, activating TPM-bypass! [ 3.421778] ima: Allocated hash algorithm: sha1 [ 3.423666] ima: No architecture policies found [ 3.425548] evm: Initialising EVM extended attributes: [ 3.428067] evm: security.selinux [ 3.429594] evm: security.ima [ 3.430714] evm: security.capability [ 3.432653] evm: HMAC attrs: 0x1 [ 3.435347] rtc_cmos 00:05: setting system clock to 2026-08-15 00:52:42 UTC (1786755162) [ 3.442596] debug: unmapping init [mem 0xffffffffa7403000-0xffffffffa75fffff] [ 3.446672] debug: unmapping init [mem 0xffffffffa6182000-0xffffffffa6458fff] [ 3.456140] Write protecting the kernel read-only data: 28672k [ 3.459632] debug: unmapping init [mem 0xffffffffa4803000-0xffffffffa49fffff] [ 3.462848] debug: unmapping init [mem 0xffffffffa5114000-0xffffffffa51fffff] [ 3.495350] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.505388] systemd[1]: Detected virtualization kvm. [ 3.507719] systemd[1]: Detected architecture x86-64. [ 3.510070] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.533969] systemd[1]: No hostname configured. [ 3.535616] systemd[1]: Set hostname to . [ 3.537848] random: systemd: uninitialized urandom read (16 bytes read) [ 3.540463] systemd[1]: Initializing machine ID from random generator. [ 3.666688] random: systemd: uninitialized urandom read (16 bytes read) [ 3.669755] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.677387] random: systemd: uninitialized urandom read (16 bytes read) [ 3.679820] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 3.685799] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.268966] device-mapper: uevent: version 1.0.3 [ 4.271432] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.901013] random: fast init done [ 4.903807] virtio_net virtio0 ens2: renamed from eth0 [ 4.966701] scsi host0: ata_piix [ 4.980486] scsi host1: ata_piix [ 4.982041] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.984290] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.943382] random: crng init done [ 9.945433] random: 7 urandom warning(s) missed due to ratelimiting [ 9.976192] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.226492] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.827139] printk: systemd: 25 output lines suppressed due to ratelimiting [ 14.721738] SELinux: Disabled at runtime. [ 14.844630] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.870624] systemd[1]: Detected virtualization kvm. [ 14.876925] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.706389] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.712385] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.724908] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.735305] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.743067] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.752428] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.768469] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message [ 17.070304] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 18.217527] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.049287] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.101156] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.628050] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.678294] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 24.684799] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit)[ 25.261376] NFS: Registering the id_resolver key type [ 25.263702] Key type id_resolver registered [ 25.265626] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg306-server login: [ 77.817258] libcfs: loading out-of-tree module taints kernel. [ 77.878858] Key type ._llcrypt registered [ 77.881680] Key type .llcrypt registered [ 78.017067] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 96.084853] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 97.582833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 97.622391] alg: No test for adler32 (adler32-zlib) [ 99.097095] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 100.009572] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 101.847516] Key type lgssc registered [ 103.905448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 125.074653] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 164.143071] hrtimer: interrupt took 12059854 ns [ 167.260500] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 180.487987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 180.533152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.795585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 181.831467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 181.979722] Lustre: lustre-MDT0000: new disk, initializing [ 182.071372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.087157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 186.667516] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.209117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.292454] Lustre: 6511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 200.323604] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 200.328718] Lustre: Skipped 1 previous similar message [ 200.412117] Lustre: lustre-MDT0001: new disk, initializing [ 200.478627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 200.502281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 200.522808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 204.135968] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 208.830477] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 218.631751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.052517] Lustre: lustre-OST0000: new disk, initializing [ 219.070858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 219.081937] Lustre: 8446:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.166051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.062434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 223.081859] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 223.153610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 225.706319] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.749560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.883381] Lustre: lustre-OST0001: new disk, initializing [ 239.886646] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 239.891485] Lustre: 9521:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 239.969419] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 245.839375] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 249.863900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 249.872288] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 249.977254] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 258.826526] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 268.927736] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 276.460806] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing check_logdir /tmp/testlogs/ [ 282.007617] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing yml_node [ 287.033284] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 289.056081] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 291.129138] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 292.606520] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 14 20:57:30 EDT 2026 [ 309.941474] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 311.355228] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 328.671945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 328.680459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 328.688179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.747631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 331.748292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 333.647844] Lustre: server umount lustre-MDT0000 complete [ 337.763074] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755496 with bad export cookie 7465862082316819761 [ 337.777621] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 341.988809] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 341.990219] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 341.992710] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 341.992721] Lustre: Skipped 3 previous similar messages [ 342.028027] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 342.044552] Lustre: Skipped 3 previous similar messages [ 343.008097] LustreError: 8638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.037203] LustreError: 8638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 347.107949] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 347.108964] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 347.129675] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 347.132750] Lustre: Skipped 4 previous similar messages [ 352.223279] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 352.224114] LustreError: 12464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 352.224161] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 352.255115] LustreError: 12464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 352.516808] Lustre: server umount lustre-MDT0001 complete [ 356.370502] Lustre: server umount lustre-OST0000 complete [ 360.255527] Lustre: server umount lustre-OST0001 complete [ 367.542995] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 376.640514] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 415.986314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 416.321315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 416.362457] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 416.495772] Lustre: lustre-MDT0000: new disk, initializing [ 416.624261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.670866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 421.838330] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 434.566844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 434.701455] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 434.752131] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 434.756501] Lustre: Skipped 1 previous similar message [ 434.859646] Lustre: lustre-MDT0001: new disk, initializing [ 434.984639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 435.013333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 435.019861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 439.161794] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 448.000938] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 453.628847] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 460.832428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 461.236640] Lustre: lustre-OST0000: new disk, initializing [ 461.242716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 461.249276] Lustre: 17046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 461.373834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.843904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 462.864332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 463.034381] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 467.614545] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 476.807569] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 481.331953] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 481.562781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 486.532923] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 486.733775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 488.932032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 488.951910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 488.960231] Lustre: Skipped 1 previous similar message [ 493.538230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 493.543398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 494.832673] Lustre: server umount lustre-OST0000 complete [ 504.171752] Lustre: server umount lustre-MDT0000 complete [ 504.294235] LustreError: 15083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.308871] LustreError: 15083:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 507.978055] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755667 with bad export cookie 7465862082316821294 [ 507.981063] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 507.988779] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 508.460850] Lustre: server umount lustre-MDT0001 complete [ 511.958591] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 21:01:09 (1786755669) [ 517.801401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 518.100064] LustreError: 19322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.161421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 521.726213] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 530.266750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.428536] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 541.012143] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 546.117962] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 553.246981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 559.919822] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 568.200995] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 576.481272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 576.498184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 576.512843] Lustre: Skipped 2 previous similar messages [ 576.522154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 576.525162] Lustre: Skipped 5 previous similar messages [ 579.552930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 581.208407] Lustre: server umount lustre-OST0000 complete [ 586.722754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 591.182653] Lustre: server umount lustre-MDT0000 complete [ 594.676559] LustreError: 19302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755753 with bad export cookie 7465862082316822176 [ 594.692962] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.085566] Lustre: server umount lustre-MDT0001 complete [ 606.336227] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 609.114706] Key type lgssc unregistered [ 609.494604] LNet: 22856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.505414] LNetError: 22856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 610.536764] LNet: Removed LNI 192.168.203.106@tcp [ 611.580322] Key type .llcrypt unregistered [ 611.587211] Key type ._llcrypt unregistered [ 639.559584] Key type ._llcrypt registered [ 639.562572] Key type .llcrypt registered [ 639.684729] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 640.740892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 640.956111] alg: No test for adler32 (adler32-zlib) [ 642.033075] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 642.236700] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 643.935246] Key type lgssc registered [ 645.405867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 653.654412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 653.675905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 655.085472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 660.107199] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 665.576188] LustreError: 24271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 665.603505] LustreError: 24271:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 668.758986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 669.257910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 673.345386] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 680.934030] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 687.320337] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 693.467440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 693.884709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 699.479338] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 700.025945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 706.117373] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 720.866421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 720.872340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 720.887634] Lustre: Skipped 1 previous similar message [ 720.899558] Lustre: Skipped 1 previous similar message [ 723.112272] Lustre: server umount lustre-OST0000 complete [ 730.597177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 730.606445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 730.623219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 732.874698] Lustre: server umount lustre-MDT0000 complete [ 736.225233] LustreError: 24271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 736.244299] LustreError: 24271:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 736.521261] LustreError: 24251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786755895 with bad export cookie 16875381558077661922 [ 736.529964] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 736.535741] LustreError: 24251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 736.792117] Lustre: server umount lustre-MDT0001 complete [ 747.918979] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 750.829625] Key type lgssc unregistered [ 751.073617] LNet: 28247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.083912] LNetError: 28247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 751.103309] LNet: Removed LNI 192.168.203.106@tcp [ 751.889209] Key type .llcrypt unregistered [ 751.894582] Key type ._llcrypt unregistered [ 776.345158] Key type ._llcrypt registered [ 776.349887] Key type .llcrypt registered [ 776.546173] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 778.209319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 778.293237] alg: No test for adler32 (adler32-zlib) [ 779.657914] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 780.051239] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 781.839175] Key type lgssc registered [ 783.056687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 790.729151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 790.740410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 792.019290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 795.641602] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 802.273770] LustreError: 29660:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 802.317743] LustreError: 29660:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 802.772350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 803.084470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 806.963960] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 813.381162] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 818.913842] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 831.597088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 831.939357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 836.856567] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 837.054413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 843.784661] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 857.068112] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 857.073587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 857.086656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 858.601276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 858.610207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 858.615595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 860.713630] Lustre: server umount lustre-OST0000 complete [ 864.737024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 864.747196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 864.767190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 864.775930] Lustre: Skipped 1 previous similar message [ 868.833475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 868.835722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 868.868021] Lustre: Skipped 1 previous similar message [ 870.184401] Lustre: server umount lustre-MDT0000 complete [ 873.456257] LustreError: 29641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756032 with bad export cookie 6962845306280998076 [ 873.472406] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 873.474383] LustreError: 29641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 873.850177] Lustre: server umount lustre-MDT0001 complete [ 883.031302] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 886.049808] Key type lgssc unregistered [ 886.374774] LNet: 33540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.384867] LNetError: 33540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.408146] LNet: Removed LNI 192.168.203.106@tcp [ 887.291440] Key type .llcrypt unregistered [ 887.301683] Key type ._llcrypt unregistered [ 911.381676] Key type ._llcrypt registered [ 911.387527] Key type .llcrypt registered [ 911.495681] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 912.348975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 912.424399] alg: No test for adler32 (adler32-zlib) [ 913.530075] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 913.728794] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 915.359187] Key type lgssc registered [ 916.298699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 923.702204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 923.736587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 925.134817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 928.983120] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 935.402113] LustreError: 34953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 935.431105] LustreError: 34953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 936.935268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.397299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 941.419476] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 949.273244] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 954.764903] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 960.767650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 961.141866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 967.818363] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 970.366307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 976.306811] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 986.081982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 986.089555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 986.099205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 986.104899] Lustre: Skipped 1 previous similar message [ 991.202833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 991.209686] Lustre: Skipped 2 previous similar messages [ 991.436987] Lustre: server umount lustre-OST0000 complete [ 996.322955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 996.333468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 996.336080] Lustre: Skipped 1 previous similar message [ 996.354214] Lustre: Skipped 1 previous similar message [ 1001.443056] LustreError: 34948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1001.471225] LustreError: 34948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1003.226588] Lustre: server umount lustre-MDT0000 complete [ 1006.447695] LustreError: 34933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756165 with bad export cookie 6064462240260681485 [ 1006.451826] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1006.455411] LustreError: 34933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1006.762335] Lustre: server umount lustre-MDT0001 complete [ 1016.222346] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1018.958796] Key type lgssc unregistered [ 1019.178645] LNet: 38489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.186095] LNetError: 38489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.204527] LNet: Removed LNI 192.168.203.106@tcp [ 1020.132849] Key type .llcrypt unregistered [ 1020.136256] Key type ._llcrypt unregistered [ 1044.387322] Key type ._llcrypt registered [ 1044.389826] Key type .llcrypt registered [ 1044.487863] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1045.195070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1045.253143] alg: No test for adler32 (adler32-zlib) [ 1046.276495] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 1046.509236] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1048.159247] Key type lgssc registered [ 1049.232367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1055.547849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1055.575510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1056.989651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1060.544826] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1067.489377] LustreError: 39899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1067.504446] LustreError: 39899:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1067.905695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.298873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1071.786558] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1078.356339] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1084.675960] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1091.693887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1092.137930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1097.668703] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1102.381801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1104.167583] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1112.322633] Lustre: lustre-OST0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 1112.553871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1112.560592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1112.594085] Lustre: Skipped 1 previous similar message [ 1117.205674] Lustre: server umount lustre-OST0000 complete [ 1140.194293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1140.202772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1140.213232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1140.224738] Lustre: Skipped 2 previous similar messages [ 1141.216624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.218229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1141.247522] Lustre: Skipped 1 previous similar message [ 1142.769875] Lustre: server umount lustre-MDT0000 complete [ 1146.337853] LustreError: 39896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1146.363553] LustreError: 39896:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1146.393438] LustreError: 39881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756305 with bad export cookie 5641833962934320112 [ 1146.405738] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1146.406053] LustreError: 39881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1152.953413] Lustre: server umount lustre-MDT0001 complete [ 1164.972922] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1167.166496] Key type lgssc unregistered [ 1167.388659] LNet: 43613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1167.392470] LNetError: 43613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1167.406851] LNet: Removed LNI 192.168.203.106@tcp [ 1168.058244] Key type .llcrypt unregistered [ 1168.060141] Key type ._llcrypt unregistered [ 1190.145221] Key type ._llcrypt registered [ 1190.152684] Key type .llcrypt registered [ 1190.307587] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1191.344192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1191.364318] alg: No test for adler32 (adler32-zlib) [ 1192.481468] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 1192.760271] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1194.447158] Key type lgssc registered [ 1195.528114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1201.952277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1201.976625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.340381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1207.185271] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1213.922973] LustreError: 45026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.936913] LustreError: 45026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1214.565385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1215.031652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1219.123318] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1225.445295] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1230.467916] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1236.358707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1236.661317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1241.791029] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1248.139683] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1248.975847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1258.734676] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (stopping) [ 1259.491745] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1259.502075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1259.519790] Lustre: Skipped 2 previous similar messages [ 1259.539394] Lustre: Skipped 2 previous similar messages [ 1260.104943] Lustre: server umount lustre-MDT0000 complete [ 1263.545487] LustreError: 45008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756422 with bad export cookie 1720192556824914693 [ 1263.558648] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1263.876905] Lustre: server umount lustre-MDT0001 complete [ 1275.875893] LustreError: 45008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.6@tcp arrived at 1786756434 with bad export cookie 1720192556824915358 [ 1275.875893] LustreError: 45007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.6@tcp arrived at 1786756434 with bad export cookie 1720192556824915358 [ 1275.875903] LustreError: 45007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1275.887500] LustreError: 45008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1295.115374] Lustre: server umount lustre-OST0000 complete [ 1310.890519] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1313.649927] Key type lgssc unregistered [ 1313.910835] LNet: 48891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.921982] LNetError: 48891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.940187] LNet: Removed LNI 192.168.203.106@tcp [ 1314.744375] Key type .llcrypt unregistered [ 1314.747221] Key type ._llcrypt unregistered [ 1340.177403] Key type ._llcrypt registered [ 1340.182818] Key type .llcrypt registered [ 1340.339982] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1341.663526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1341.846634] alg: No test for adler32 (adler32-zlib) [ 1342.969565] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 1343.220231] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1344.919557] Key type lgssc registered [ 1346.386275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1353.829332] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1353.853162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1369.439416] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1374.559537] LustreError: 50285:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 1401.183289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1407.184272] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1415.685788] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1582.499094] Lustre: server umount lustre-OST0000 complete [ 1598.437520] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1601.170683] Key type lgssc unregistered [ 1601.441656] LNet: 51968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.449427] LNetError: 51968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1601.465634] LNet: Removed LNI 192.168.203.106@tcp [ 1602.299229] Key type .llcrypt unregistered [ 1602.308582] Key type ._llcrypt unregistered [ 1626.284436] Key type ._llcrypt registered [ 1626.289589] Key type .llcrypt registered [ 1626.462872] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1628.004511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1628.036807] alg: No test for adler32 (adler32-zlib) [ 1629.120914] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 1629.430843] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1631.145678] Key type lgssc registered [ 1632.381701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1640.518151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1640.542788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1641.991497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1646.822651] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1652.197451] LustreError: 53383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1652.219736] LustreError: 53383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1653.217675] LustreError: 53382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1655.430395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1655.863491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1660.624829] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1668.480964] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1674.402774] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1681.238297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1681.549849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1686.814886] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1688.750942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1694.755332] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1699.296105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1699.309249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1699.330519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1700.386935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1703.474300] Lustre: server umount lustre-OST0000 complete [ 1710.562165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.573288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1710.592523] Lustre: Skipped 1 previous similar message [ 1710.623079] Lustre: Skipped 2 previous similar messages [ 1713.596398] Lustre: server umount lustre-MDT0000 complete [ 1715.691271] LustreError: 53382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1715.713825] LustreError: 53382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1718.117425] LustreError: 53361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786756877 with bad export cookie 11329363449355829358 [ 1718.127106] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1718.139892] LustreError: 53361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1718.605683] Lustre: server umount lustre-MDT0001 complete [ 1731.160722] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1733.959644] Key type lgssc unregistered [ 1734.237350] LNet: 56914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1734.256693] LNetError: 56914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1734.275037] LNet: Removed LNI 192.168.203.106@tcp [ 1735.101644] Key type .llcrypt unregistered [ 1735.103985] Key type ._llcrypt unregistered [ 1759.644883] Key type ._llcrypt registered [ 1759.648165] Key type .llcrypt registered [ 1759.747077] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1760.732144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1760.941121] alg: No test for adler32 (adler32-zlib) [ 1762.083591] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 1762.503903] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1764.319593] Key type lgssc registered [ 1766.031121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1773.911251] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1773.929154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1789.535451] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1794.657550] LustreError: 58301:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 1821.343360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1826.643566] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1834.457366] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1842.688910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1843.298437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1843.370462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1848.291798] LustreError: 58326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1848.309434] LustreError: 58326:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1848.407991] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1853.414531] LustreError: 58327:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1856.482445] LustreError: 59339:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1856.539692] LustreError: 59339:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1857.374650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1857.784269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1863.367667] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1872.011596] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1878.011935] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1883.618528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1883.630129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1883.659168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1883.667581] Lustre: Skipped 1 previous similar message [ 1886.885704] Lustre: server umount lustre-OST0000 complete [ 1913.313144] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1913.346609] Lustre: Skipped 1 previous similar message [ 1913.380700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1913.394228] Lustre: Skipped 1 previous similar message [ 1917.730446] Lustre: server umount lustre-MDT0000 complete [ 1918.436179] LustreError: 59343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1918.466232] LustreError: 59343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1922.191877] LustreError: 58307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757081 with bad export cookie 14110655014641004502 [ 1922.207332] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1922.223159] LustreError: 58307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1928.649795] Lustre: server umount lustre-MDT0001 complete [ 1939.026708] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 1941.718096] Key type lgssc unregistered [ 1942.014644] LNet: 62050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1942.019866] LNetError: 62050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1942.036528] LNet: Removed LNI 192.168.203.106@tcp [ 1942.796587] Key type .llcrypt unregistered [ 1942.798863] Key type ._llcrypt unregistered [ 1968.030819] Key type ._llcrypt registered [ 1968.034342] Key type .llcrypt registered [ 1968.115981] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 1969.355303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1969.488790] alg: No test for adler32 (adler32-zlib) [ 1970.508796] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 1970.827501] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 1972.495276] Key type lgssc registered [ 1973.688581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1982.277774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1982.303694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.868479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1988.780254] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 1993.188989] LustreError: 63462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1993.246178] LustreError: 63462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1998.316896] LustreError: 63461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1998.846232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1999.317642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2004.305934] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2012.590920] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2018.896705] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2028.285828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2028.749952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2038.058305] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2046.514599] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2053.363904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2074.082729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2074.111288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2077.760788] Lustre: server umount lustre-OST0000 complete [ 2084.335073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.337538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2084.361248] Lustre: Skipped 2 previous similar messages [ 2084.366717] Lustre: Skipped 1 previous similar message [ 2088.135852] Lustre: server umount lustre-MDT0000 complete [ 2089.440563] LustreError: 63462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2089.482902] LustreError: 63462:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2091.942841] LustreError: 63442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757251 with bad export cookie 2943880166057666074 [ 2091.947435] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2091.956872] LustreError: 63442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2092.404108] Lustre: server umount lustre-MDT0001 complete [ 2103.848919] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2106.306905] Key type lgssc unregistered [ 2106.678865] LNet: 67022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.682823] LNetError: 67022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2106.695659] LNet: Removed LNI 192.168.203.106@tcp [ 2107.418893] Key type .llcrypt unregistered [ 2107.425842] Key type ._llcrypt unregistered [ 2131.049951] Key type ._llcrypt registered [ 2131.052439] Key type .llcrypt registered [ 2131.151469] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2132.010651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2132.177049] alg: No test for adler32 (adler32-zlib) [ 2133.179982] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 2133.459394] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2135.183156] Key type lgssc registered [ 2136.076751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2144.810870] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 21:28:22 (1786757302) [ 2150.962301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2150.991209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.513405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2157.221514] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2162.665889] LustreError: 68710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.714348] LustreError: 68710:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2167.547082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.784482] LustreError: 68711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.279248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2175.108534] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2184.002508] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2190.474527] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2197.804332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2198.147819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2204.977487] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2213.907559] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2222.796407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2230.593083] Lustre: Failing over lustre-MDT0000 [ 2231.036091] Lustre: server umount lustre-MDT0000 complete [ 2233.327710] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.341862] LustreError: 69438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2233.358812] Lustre: Skipped 1 previous similar message [ 2233.397451] LustreError: 69438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2238.438533] LustreError: 68707:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.475536] LustreError: 68707:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2243.133023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2243.377027] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2243.553331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2243.557971] Lustre: *** cfs_fail_loc=135, val=0*** [ 2243.561016] Lustre: Skipped 2 previous similar messages [ 2243.672139] LustreError: 71818:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2243.683092] LustreError: 71818:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.106@tcp: cfg command failed: rc = -2 [ 2243.689547] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2243.689547] [ 2243.695500] LustreError: MGC192.168.203.106@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2243.705461] LustreError: 71807:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2243.715601] LustreError: 71807:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2243.727127] LustreError: 71807:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2243.733457] LustreError: 71807:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2243.757474] Lustre: server umount lustre-MDT0000 complete [ 2243.760473] LustreError: 71807:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2245.331654] LustreError: 68706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2245.357978] LustreError: 68706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2250.566822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2250.687549] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2250.999514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2251.047828] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2251.525470] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2254.857737] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2256.368600] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2256.445654] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2256.497832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2265.056532] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2265.064414] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.080856] Lustre: Skipped 1 previous similar message [ 2265.094726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2266.609511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2266.623270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.716967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2271.721925] Lustre: Skipped 2 previous similar messages [ 2276.836430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2276.861442] Lustre: Skipped 1 previous similar message [ 2277.343346] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2277.763541] Lustre: server umount lustre-OST0000 complete [ 2281.952506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2281.953873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2287.072410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2287.075526] Lustre: Skipped 2 previous similar messages [ 2296.287117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2296.513872] Lustre: server umount lustre-MDT0000 complete [ 2297.316979] LustreError: 68706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.341530] LustreError: 68706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2300.382579] LustreError: 72645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757459 with bad export cookie 13054800576802881886 [ 2300.388239] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2300.391117] LustreError: 72645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2300.719576] Lustre: server umount lustre-MDT0001 complete [ 2313.001112] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2315.253599] Key type lgssc unregistered [ 2315.583206] LNet: 73606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2315.587227] LNetError: 73606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2315.607876] LNet: Removed LNI 192.168.203.106@tcp [ 2316.510677] Key type .llcrypt unregistered [ 2316.512989] Key type ._llcrypt unregistered [ 2341.209649] Key type ._llcrypt registered [ 2341.212591] Key type .llcrypt registered [ 2341.368079] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2342.679595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2342.743793] alg: No test for adler32 (adler32-zlib) [ 2343.963229] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 2344.404745] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2346.207174] Key type lgssc registered [ 2347.733822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2356.603530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2356.626815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.254576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2363.334694] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2368.483856] LustreError: 75030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2368.500051] LustreError: 75030:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2372.661376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2373.202519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2379.179883] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2388.429424] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2396.337533] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2406.222299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2406.737116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2417.770467] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2427.238879] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2428.346612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2433.775817] Lustre: Failing over lustre-MDT0000 [ 2433.993550] Lustre: server umount lustre-MDT0000 complete [ 2434.531682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2434.546744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.568859] LustreError: 75030:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.628039] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2438.630203] LustreError: 75031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.654331] Lustre: Skipped 1 previous similar message [ 2438.705335] LustreError: 75031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2443.749398] LustreError: 75026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2443.779347] LustreError: 75026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2445.188648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2445.382205] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2445.546386] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (not set up) [ 2445.630349] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2445.797794] LustreError: 78122:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2445.803979] LustreError: 78122:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.106@tcp: cfg command failed: rc = -9 [ 2445.815909] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2445.815909] [ 2445.821859] LustreError: MGC192.168.203.106@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2445.834118] LustreError: 78110:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2445.844436] LustreError: 78110:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2445.853696] LustreError: 78110:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2445.859218] LustreError: 78110:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2445.881607] Lustre: server umount lustre-MDT0000 complete [ 2445.884992] LustreError: 78110:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2448.872182] LustreError: 75031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2448.918944] LustreError: 75031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2455.466453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2455.606872] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2455.620516] LustreError: 75022:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2455.626851] LustreError: 78469:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.203.106@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000063e53d44) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2455.768324] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.6@tcp (not set up) [ 2455.886872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2455.944819] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2460.881155] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2460.972537] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2461.219391] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2461.265270] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2461.350728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2469.344803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2469.349718] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2469.361489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2471.394379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2471.411756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2474.940782] Lustre: server umount lustre-OST0000 complete [ 2481.635215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2481.648855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2481.658357] Lustre: Skipped 2 previous similar messages [ 2485.115837] Lustre: server umount lustre-MDT0000 complete [ 2486.756537] LustreError: 75031:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.772243] LustreError: 75031:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2488.662517] LustreError: 75011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757647 with bad export cookie 10391214133444866212 [ 2488.665239] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2488.671327] LustreError: 75011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2488.937905] Lustre: server umount lustre-MDT0001 complete [ 2501.255862] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2504.288604] Key type lgssc unregistered [ 2504.543757] LNet: 79907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2504.550444] LNetError: 79907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2504.584630] LNet: Removed LNI 192.168.203.106@tcp [ 2505.524204] Key type .llcrypt unregistered [ 2505.528332] Key type ._llcrypt unregistered [ 2519.126405] Key type ._llcrypt registered [ 2519.128986] Key type .llcrypt registered [ 2519.235305] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2519.602984] Key type .llcrypt unregistered [ 2519.607235] Key type ._llcrypt unregistered [ 2544.415254] Key type ._llcrypt registered [ 2544.417209] Key type .llcrypt registered [ 2544.481592] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2545.475352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2545.514605] alg: No test for adler32 (adler32-zlib) [ 2546.510931] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 2546.802171] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2548.447591] Key type lgssc registered [ 2549.637835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2556.862222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2556.904531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2558.433898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.511361] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2568.684889] LustreError: 81818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.715565] LustreError: 81818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2570.754251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2571.331174] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2575.883764] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2583.370756] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2589.763477] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2596.813668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2597.213954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2603.403045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2605.145356] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2612.723110] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2627.040965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2627.050225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2627.055765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2627.070657] Lustre: Skipped 1 previous similar message [ 2632.168435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2632.178798] Lustre: Skipped 1 previous similar message [ 2633.016903] Lustre: server umount lustre-OST0000 complete [ 2637.793422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2637.809877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.852248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.870342] Lustre: Skipped 1 previous similar message [ 2642.403449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2642.408770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2643.833609] Lustre: server umount lustre-MDT0000 complete [ 2647.523709] LustreError: 81818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2647.550912] LustreError: 81818:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2647.851748] LustreError: 81798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757806 with bad export cookie 7381820028694246500 [ 2647.861072] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2647.865212] LustreError: 81798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2648.459565] Lustre: server umount lustre-MDT0001 complete [ 2662.281050] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2665.174441] Key type lgssc unregistered [ 2665.660616] LNet: 85374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2665.665754] LNetError: 85374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2665.682281] LNet: Removed LNI 192.168.203.106@tcp [ 2666.703426] Key type .llcrypt unregistered [ 2666.705794] Key type ._llcrypt unregistered [ 2691.584913] Key type ._llcrypt registered [ 2691.591161] Key type .llcrypt registered [ 2691.752270] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2693.158344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2693.249367] alg: No test for adler32 (adler32-zlib) [ 2694.369807] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 2694.654215] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2696.343942] Key type lgssc registered [ 2698.282445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2707.383500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2707.420816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2709.003608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2713.800453] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2719.201860] LustreError: 86787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2719.223566] LustreError: 86787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2724.162102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2724.340149] LustreError: 86788:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.659165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2729.433859] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2737.990140] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2743.677155] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2749.679223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2750.176868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2754.292752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2757.734786] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2765.606921] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2780.129967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2780.132321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2780.151809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2780.160246] Lustre: Skipped 1 previous similar message [ 2785.251432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2785.263948] Lustre: Skipped 2 previous similar messages [ 2790.367188] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2790.369454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2790.399845] Lustre: Skipped 1 previous similar message [ 2790.621970] Lustre: server umount lustre-OST0000 complete [ 2795.495725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2795.498759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2795.515545] Lustre: Skipped 1 previous similar message [ 2795.537187] Lustre: Skipped 1 previous similar message [ 2800.403919] Lustre: server umount lustre-MDT0000 complete [ 2800.618238] LustreError: 86787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2800.631143] LustreError: 86787:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2803.684245] LustreError: 86768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786757962 with bad export cookie 10984038310304611329 [ 2803.696879] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2804.071942] Lustre: server umount lustre-MDT0001 complete [ 2814.280700] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2816.561876] Key type lgssc unregistered [ 2816.961872] LNet: 90323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2816.973302] LNetError: 90323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2817.005944] LNet: Removed LNI 192.168.203.106@tcp [ 2817.864436] Key type .llcrypt unregistered [ 2817.866833] Key type ._llcrypt unregistered [ 2840.155620] Key type ._llcrypt registered [ 2840.161258] Key type .llcrypt registered [ 2840.262882] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2841.359248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2841.472237] alg: No test for adler32 (adler32-zlib) [ 2842.600714] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 2842.952557] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2844.623810] Key type lgssc registered [ 2846.068090] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2853.754706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2853.781901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2855.254655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2859.271891] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2865.636040] LustreError: 91737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2865.655297] LustreError: 91737:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2867.591138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.917709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2872.958359] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2879.684671] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2885.093994] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2891.332958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2891.692791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2897.635828] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 2900.922142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2905.264501] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2918.897486] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2918.909658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2918.937866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2921.455288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2921.458137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2924.957876] Lustre: server umount lustre-OST0000 complete [ 2929.638743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2929.657910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2929.688732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2929.691868] Lustre: Skipped 1 previous similar message [ 2931.681317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2931.716383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2931.721688] Lustre: Skipped 1 previous similar message [ 2935.002566] Lustre: server umount lustre-MDT0000 complete [ 2936.804437] LustreError: 91737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2936.817767] LustreError: 91737:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2938.564497] LustreError: 91718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758097 with bad export cookie 17337862639860229628 [ 2938.582653] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2938.947340] Lustre: server umount lustre-MDT0001 complete [ 2949.949183] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 2952.514864] Key type lgssc unregistered [ 2952.836908] LNet: 95293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2952.845203] LNetError: 95293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2952.867825] LNet: Removed LNI 192.168.203.106@tcp [ 2953.742969] Key type .llcrypt unregistered [ 2953.745563] Key type ._llcrypt unregistered [ 2979.422343] Key type ._llcrypt registered [ 2979.430804] Key type .llcrypt registered [ 2979.603874] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 2981.446135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2981.533135] alg: No test for adler32 (adler32-zlib) [ 2982.833872] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 2983.176137] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 2984.872994] Key type lgssc registered [ 2985.926436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3000.120904] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3000.163291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3016.351503] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3021.474192] LustreError: 96681:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 3048.159364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3054.229352] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3063.273179] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3069.286423] Lustre: server umount lustre-OST0000 complete [ 3078.061959] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 21:43:55 (1786758235) [ 3084.726517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3085.330344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3089.333133] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3090.407496] LustreError: 98321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3090.435717] LustreError: 98321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3095.523667] LustreError: 98322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3098.387494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3098.922523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3103.637446] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3111.342885] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3116.804572] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3123.835574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3124.144692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3129.959691] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3130.289564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3136.682118] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3153.910467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3153.915785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3153.934956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3153.956533] Lustre: Skipped 1 previous similar message [ 3157.900660] Lustre: server umount lustre-OST0000 complete [ 3164.133210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3164.156690] Lustre: Skipped 1 previous similar message [ 3164.163447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3164.169287] Lustre: Skipped 1 previous similar message [ 3168.132765] Lustre: server umount lustre-MDT0000 complete [ 3169.248299] LustreError: 98322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3169.282376] LustreError: 98322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3172.560914] LustreError: 98303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758331 with bad export cookie 12031008740071033505 [ 3172.565719] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3172.580673] LustreError: 98303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3173.038537] Lustre: server umount lustre-MDT0001 complete [ 3184.947500] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3188.311742] Key type lgssc unregistered [ 3188.789696] LNet: 102016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3188.801968] LNetError: 102016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3188.820880] LNet: Removed LNI 192.168.203.106@tcp [ 3189.849968] Key type .llcrypt unregistered [ 3189.852372] Key type ._llcrypt unregistered [ 3231.938398] Key type ._llcrypt registered [ 3231.942690] Key type .llcrypt registered [ 3232.031905] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3232.685536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3232.712917] alg: No test for adler32 (adler32-zlib) [ 3233.703282] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 3233.902414] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3235.552925] Key type lgssc registered [ 3236.499186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3243.035947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3243.061687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3244.414843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3248.435674] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3254.760963] LustreError: 104203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3254.780524] LustreError: 104203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3257.034293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3257.264569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3261.072933] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3267.708449] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3273.168882] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3279.333886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3279.666302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3285.520765] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3289.985768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3293.485955] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3303.394487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3303.407762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3303.418322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3305.440470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3305.444310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3305.451160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.486988] Lustre: Skipped 1 previous similar message [ 3308.894596] Lustre: server umount lustre-OST0000 complete [ 3313.636772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3313.645925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3313.668248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3315.684541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3315.694488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3319.361766] Lustre: server umount lustre-MDT0000 complete [ 3320.803228] LustreError: 104204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3320.817817] LustreError: 104204:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3323.405786] LustreError: 104185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758482 with bad export cookie 9031783203424000023 [ 3323.418137] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3323.422544] LustreError: 104185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3323.769526] Lustre: server umount lustre-MDT0001 complete [ 3333.825988] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3336.210207] Key type lgssc unregistered [ 3336.441547] LNet: 107739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3336.451274] LNetError: 107739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3337.509869] LNet: Removed LNI 192.168.203.106@tcp [ 3338.340727] Key type .llcrypt unregistered [ 3338.342541] Key type ._llcrypt unregistered [ 3356.797294] Key type ._llcrypt registered [ 3356.809728] Key type .llcrypt registered [ 3356.912468] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3357.819531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3357.845074] alg: No test for adler32 (adler32-zlib) [ 3358.938695] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 3359.195052] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3360.903169] Key type lgssc registered [ 3362.441608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3370.569290] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3370.584209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3386.079584] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3391.200249] LustreError: 108980:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 3417.823651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3423.404179] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3431.635379] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3439.266868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3439.515621] LustreError: MGC192.168.203.106@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3439.531626] LustreError: 110004:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3439.542794] LustreError: 110004:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3439.555703] LustreError: 110004:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3439.678213] Lustre: server umount lustre-MDT0000 complete [ 3439.681495] LustreError: 110004:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3460.063196] Lustre: 108317:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786758603/real 1786758603] req@000000000d60599d x1873552099773696/t0(0) o400->MGC192.168.203.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786758619 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3460.105033] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3460.113513] LustreError: 108992:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3480.681358] Lustre: server umount lustre-OST0000 complete [ 3488.647828] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 3498.172535] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3530.874218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3531.080576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3531.105378] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3531.172207] Lustre: lustre-MDT0000: new disk, initializing [ 3531.235446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3531.253361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3534.658976] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3546.196873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3546.396399] Lustre: 112798:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3546.454342] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3546.460319] Lustre: Skipped 1 previous similar message [ 3546.568906] Lustre: lustre-MDT0001: new disk, initializing [ 3546.708844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3546.756018] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3546.781556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3551.438287] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3560.701976] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3566.588295] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3573.401989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3573.646480] Lustre: lustre-OST0000: new disk, initializing [ 3573.650331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3573.655169] Lustre: 114768:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3573.730664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3579.436089] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3581.516106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3581.526324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3581.620468] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3588.340310] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3592.737390] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3592.946460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3597.321392] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3597.491383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3602.402782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3602.416857] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3602.421283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3602.445418] LustreError: Skipped 1 previous similar message [ 3605.461478] Lustre: server umount lustre-OST0000 complete [ 3612.645780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3612.651549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3612.664146] Lustre: Skipped 2 previous similar messages [ 3612.669741] Lustre: Skipped 1 previous similar message [ 3617.763821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3617.775203] Lustre: Skipped 2 previous similar messages [ 3622.880377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3622.882026] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 3622.895309] Lustre: Skipped 1 previous similar message [ 3623.104873] Lustre: server umount lustre-MDT0000 complete [ 3626.703073] LustreError: 113604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758785 with bad export cookie 8411034270320212608 [ 3626.706354] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3626.725471] LustreError: 113604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3627.157755] Lustre: server umount lustre-MDT0001 complete [ 3635.807482] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 21:53:13 (1786758793) [ 3638.728344] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3653.393394] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 3663.256286] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3693.929556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3694.192942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3694.215604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3694.311764] Lustre: lustre-MDT0000: new disk, initializing [ 3694.393295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3694.409865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3698.459577] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3707.895244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3707.984384] Lustre: 119666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3708.024636] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3708.028144] Lustre: Skipped 1 previous similar message [ 3708.077971] Lustre: lustre-MDT0001: new disk, initializing [ 3708.169275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3708.205065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3708.214133] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3712.109527] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3721.735824] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3727.651905] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3734.281535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3734.540372] Lustre: lustre-OST0000: new disk, initializing [ 3734.543602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3734.551264] Lustre: 121634:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3734.632446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3740.741956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3740.755768] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3740.931912] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3741.518760] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3749.940837] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3755.101469] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3755.383905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3759.820326] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3759.972647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3766.757649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3766.757890] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3766.771758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3766.778975] LustreError: Skipped 1 previous similar message [ 3768.138168] Lustre: server umount lustre-OST0000 complete [ 3771.872651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.890972] Lustre: Skipped 2 previous similar messages [ 3777.004708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3777.009842] Lustre: Skipped 4 previous similar messages [ 3777.705161] Lustre: server umount lustre-MDT0000 complete [ 3781.339665] LustreError: 119658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786758940 with bad export cookie 8411034270320213483 [ 3781.343129] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3781.347103] LustreError: 119658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3781.700302] Lustre: server umount lustre-MDT0001 complete [ 3792.166906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3793.142046] LustreError: 123853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3793.192634] LustreError: 123853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3793.337975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3798.499911] LustreError: 123854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3799.662299] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3803.629051] LustreError: 123853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3808.736721] LustreError: 123854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3809.776168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3815.407748] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3822.474901] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3828.012472] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3833.856921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3839.763473] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3847.152403] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3854.394924] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3858.915547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3858.933984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3865.114305] Lustre: server umount lustre-OST0000 complete [ 3871.714476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3871.736152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3871.760641] Lustre: Skipped 1 previous similar message [ 3875.522450] Lustre: server umount lustre-MDT0000 complete [ 3878.834333] LustreError: 123833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759037 with bad export cookie 8411034270320214372 [ 3878.838190] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3878.848186] LustreError: 123833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3879.207548] Lustre: server umount lustre-MDT0001 complete [ 3888.850261] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 3891.296210] Key type lgssc unregistered [ 3891.626113] LNet: 127487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3891.634092] LNetError: 127487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3891.653639] LNet: Removed LNI 192.168.203.106@tcp [ 3892.354490] Key type .llcrypt unregistered [ 3892.361618] Key type ._llcrypt unregistered [ 3908.079420] Key type ._llcrypt registered [ 3908.081285] Key type .llcrypt registered [ 3908.182731] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 3921.369287] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 3922.322617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3922.463285] alg: No test for adler32 (adler32-zlib) [ 3923.566913] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 3923.847241] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 3925.487179] Key type lgssc registered [ 3926.541752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3961.991661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3962.009925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3963.350072] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3963.381259] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3963.510102] Lustre: lustre-MDT0000: new disk, initializing [ 3963.570069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3963.595404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3967.440430] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3977.424882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3977.576336] Lustre: 130648:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3977.653626] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3977.659375] Lustre: Skipped 1 previous similar message [ 3977.778563] Lustre: lustre-MDT0001: new disk, initializing [ 3977.872543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3977.909612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3977.931346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3982.286200] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 3991.192694] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3997.167778] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4003.282241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4003.497441] Lustre: lustre-OST0000: new disk, initializing [ 4003.503605] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4003.511632] Lustre: 132616:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4003.609771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4009.180870] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4010.051618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4010.065952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4010.264966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4017.976793] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4022.007464] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4022.206970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4026.293169] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4026.521916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4030.956606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4030.964135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4030.966981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4030.978874] Lustre: Skipped 1 previous similar message [ 4034.467844] Lustre: server umount lustre-OST0000 complete [ 4039.649680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4039.663436] LustreError: Skipped 1 previous similar message [ 4039.676141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4039.692625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4039.697735] Lustre: Skipped 1 previous similar message [ 4041.194235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4041.201860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4044.583375] Lustre: server umount lustre-MDT0000 complete [ 4048.738827] LustreError: 131443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759207 with bad export cookie 6308906386884069689 [ 4048.750282] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4048.754779] LustreError: 131443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4049.306125] Lustre: server umount lustre-MDT0001 complete [ 4057.635054] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 22:00:15 (1786759215) [ 4063.529634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4064.021187] LustreError: 135027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4064.041491] LustreError: 135027:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4064.126509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4068.225213] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4069.361305] LustreError: 135028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4074.470509] LustreError: 135027:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4076.490046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4076.812649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4080.568768] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4087.751455] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4093.149788] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4097.512169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4097.530659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4097.555194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4097.557866] Lustre: Skipped 1 previous similar message [ 4106.209745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4106.214485] Lustre: Skipped 5 previous similar messages [ 4109.279140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4109.572905] Lustre: server umount lustre-MDT0000 complete [ 4111.329943] LustreError: 135028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.348746] LustreError: 135028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4112.773376] LustreError: 135009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759271 with bad export cookie 6308906386884070571 [ 4112.781149] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4112.783983] LustreError: 135009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4113.057543] Lustre: server umount lustre-MDT0001 complete [ 4120.640272] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 22:01:18 (1786759278) [ 4126.365211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4141.919917] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4147.039493] LustreError: 137454:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 4172.767655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4179.555584] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4187.148646] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4189.262781] Lustre: server umount lustre-OST0000 complete [ 4197.874936] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 22:02:35 (1786759355) [ 4204.743100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4205.143115] LustreError: 138901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4205.212301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4210.457752] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4215.780907] LustreError: 138902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4215.806714] LustreError: 138902:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4219.568191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4223.834145] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4231.840557] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4238.413909] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4244.610846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4244.952821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4244.964056] Lustre: Skipped 1 previous similar message [ 4250.801711] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4258.235769] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4271.076423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4271.084888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4271.099730] Lustre: Skipped 1 previous similar message [ 4271.103322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4276.929682] Lustre: server umount lustre-MDT0000 complete [ 4279.779162] LustreError: 138896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4279.792403] LustreError: 138896:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4280.204484] LustreError: 138883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759439 with bad export cookie 6308906386884071236 [ 4280.213965] LustreError: 138883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4280.225252] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4280.494322] Lustre: server umount lustre-MDT0001 complete [ 4294.204970] Lustre: server umount lustre-OST0000 complete [ 4302.698917] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 22:04:20 (1786759460) [ 4308.850478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4309.539628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4313.818952] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4314.601322] LustreError: 142570:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4314.620464] LustreError: 142570:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4322.629294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4328.173909] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4336.415807] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4342.708220] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4348.737099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4354.262924] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4355.092737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4362.508424] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4366.558594] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4366.770860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4371.457768] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4371.710120] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4374.497164] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4374.509106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4374.527674] Lustre: Skipped 2 previous similar messages [ 4374.531153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4374.541264] Lustre: Skipped 3 previous similar messages [ 4379.506294] Lustre: server umount lustre-OST0000 complete [ 4384.736344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4384.745777] LustreError: Skipped 1 previous similar message [ 4389.325798] Lustre: server umount lustre-MDT0000 complete [ 4390.884619] LustreError: 142569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4390.898823] LustreError: 142569:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4392.836500] LustreError: 142549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759551 with bad export cookie 6308906386884072307 [ 4392.852330] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4392.856018] LustreError: 142549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4393.122953] Lustre: server umount lustre-MDT0001 complete [ 4401.009806] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 22:05:58 (1786759558) [ 4406.802862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4422.369165] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4427.489194] LustreError: 146724:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 4453.152058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4453.162758] Lustre: Skipped 2 previous similar messages [ 4460.827333] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4471.150828] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4479.046784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4479.468917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4484.378954] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4492.367141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4497.128780] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4504.116789] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4511.217772] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4517.846556] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4518.308833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4524.695612] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4525.031890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4528.608691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4528.615085] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4528.623270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4528.623279] Lustre: Skipped 5 previous similar messages [ 4528.664353] Lustre: Skipped 5 previous similar messages [ 4534.283597] Lustre: server umount lustre-MDT0000 complete [ 4538.849029] LustreError: 147761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4538.876950] LustreError: 147761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4539.277987] LustreError: 146730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759698 with bad export cookie 6308906386884073329 [ 4539.282566] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4539.300423] LustreError: 146730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4539.673466] Lustre: server umount lustre-MDT0001 complete [ 4552.208622] Lustre: server umount lustre-OST0000 complete [ 4560.471489] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 22:08:38 (1786759718) [ 4567.142615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4582.688258] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4587.809514] LustreError: 150919:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 4619.443597] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4626.179724] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4632.806167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4633.387460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4637.676533] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4648.534748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4653.710144] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4661.463699] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4667.908669] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4674.821471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4674.960441] Lustre: lustre-OST0001: new disk, initializing [ 4674.965458] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4674.976862] Lustre: 153790:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4681.385061] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4685.404176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4685.427730] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4685.590437] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4690.962755] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4695.576375] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4695.797915] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4700.271752] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4700.496902] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4705.263444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4705.278813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4705.310560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4705.319494] Lustre: Skipped 6 previous similar messages [ 4708.363231] Lustre: server umount lustre-OST0000 complete [ 4718.459057] Lustre: server umount lustre-OST0001 complete [ 4729.046168] Lustre: server umount lustre-MDT0000 complete [ 4732.917893] LustreError: 150927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786759891 with bad export cookie 6308906386884074281 [ 4732.926484] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4732.937722] LustreError: 150927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4733.270891] Lustre: server umount lustre-MDT0001 complete [ 4758.363112] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 22:11:55 (1786759915) [ 4760.988843] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 4763.032722] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 4764.813341] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 22:12:02 (1786759922) [ 4773.503971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4773.750816] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4773.818865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4774.044711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4774.049473] Lustre: Skipped 6 previous similar messages [ 4779.382783] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4789.005947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4789.222140] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4789.275740] Lustre: 157351:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4794.955989] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4804.593432] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4810.307879] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4816.758470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4817.028145] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4823.807812] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4827.272365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 4832.206534] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4836.566589] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4836.718120] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4841.707958] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4841.989751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4847.588590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4847.600179] LustreError: Skipped 4 previous similar messages [ 4847.604369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4847.621618] Lustre: Skipped 5 previous similar messages [ 4847.626280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4847.638678] Lustre: Skipped 7 previous similar messages [ 4858.335194] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4858.627612] Lustre: server umount lustre-OST0000 complete [ 4882.055038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4888.802382] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4888.822118] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4888.827827] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4889.950535] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 4898.484719] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4903.755833] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4903.974716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4908.107859] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4908.338838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4914.114755] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a4c46a57800.ost_server_uuid 50 [ 4915.682486] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8a4c46a57800.ost_server_uuid in FULL state after 0 sec [ 4916.051765] LustreError: 157357:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.203.6@tcp) returned error from blocking AST (req@000000006ca33e98 x1873552691432448 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000e102ee86/0x578dbceecf6b2483 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.6@tcp remote: 0x2934103e7b805540 expref: 5 pid: 157359 timeout: 5016 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4916.106147] LustreError: lustre-MDT0000: A client on nid 192.168.203.6@tcp was evicted due to a lock blocking callback time out: rc -107 [ 4916.114125] LustreError: 157347:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.203.6@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000e102ee86/0x578dbceecf6b2483 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.203.6@tcp remote: 0x2934103e7b805540 expref: 6 pid: 157359 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 4926.260970] Lustre: server umount lustre-OST0000 complete [ 4944.351357] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4944.596416] Lustre: server umount lustre-MDT0000 complete [ 4945.380070] LustreError: 158432:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4945.393170] LustreError: 158432:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 4947.780714] LustreError: 157342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760106 with bad export cookie 6308906386884075464 [ 4947.782309] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4947.788190] LustreError: 157342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4962.271161] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4962.501744] Lustre: server umount lustre-MDT0001 complete [ 4974.234272] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 4976.592369] Key type lgssc unregistered [ 4976.823785] LNet: 163474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4976.834300] LNetError: 163474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4976.851728] LNet: Removed LNI 192.168.203.106@tcp [ 4977.568598] Key type .llcrypt unregistered [ 4977.570707] Key type ._llcrypt unregistered [ 4999.963644] Key type ._llcrypt registered [ 4999.965795] Key type .llcrypt registered [ 5000.064028] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5001.157934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5001.205294] alg: No test for adler32 (adler32-zlib) [ 5002.308374] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 5002.584307] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 5004.303215] Key type lgssc registered [ 5005.386244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5012.575785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5012.594163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5014.162526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5019.043956] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5024.746422] LustreError: 164891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5024.770467] LustreError: 164891:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5027.843537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5028.183722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5032.574423] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5041.911120] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5049.053286] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5058.598345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5058.991958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5060.050557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5065.148830] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5073.820069] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5080.102920] Lustre: Failing over lustre-MDT0000 [ 5080.638981] Lustre: server umount lustre-MDT0000 complete [ 5080.841302] LustreError: 166787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5083.617631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5083.626375] LustreError: 164887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5083.640173] Lustre: Skipped 2 previous similar messages [ 5083.685605] LustreError: 164887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5088.738027] LustreError: 164884:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5088.764793] LustreError: 164884:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5093.652710] LustreError: 164870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.6@tcp arrived at 1786760252 with bad export cookie 2117938471558031491 [ 5093.865533] LustreError: 164886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5093.888086] LustreError: 164886:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5098.976165] Lustre: 164198:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786760242/real 1786760242] req@00000000e6447fe8 x1873553822608640/t0(0) o400->MGC192.168.203.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1786760258 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5099.012314] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5100.369955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5104.107229] LustreError: 164887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5104.121063] LustreError: 164887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5109.219779] LustreError: 164196:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c9dcc534 x1873553822613760/t0(0) o250->MGC192.168.203.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5109.748699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5109.851086] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5114.078484] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5114.853489] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5114.855645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5116.729643] Lustre: lustre-MDT0000: Denying connection for new client 816bb538-cf3f-4e29-b071-5759c1ef2acc (at 192.168.203.6@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5121.745728] Lustre: lustre-MDT0000: Denying connection for new client 816bb538-cf3f-4e29-b071-5759c1ef2acc (at 192.168.203.6@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:02 [ 5129.711075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5129.720274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5129.733917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5132.478755] Lustre: server umount lustre-OST0000 complete [ 5137.576771] LustreError: 168624:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5137.606535] Lustre: 167972:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5137.630837] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5137.666070] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5137.685096] LustreError: 167972:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000086d8fbcd x1873553822629248/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5137.736303] LustreError: 167972:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5137.757463] LustreError: 167972:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5137.774938] Lustre: lustre-MDT0000: Recovery over after 0:23, of 2 clients 0 recovered and 2 were evicted. [ 5138.180850] Lustre: server umount lustre-MDT0000 complete [ 5140.467187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5140.481985] Lustre: Skipped 1 previous similar message [ 5140.491658] LustreError: 164886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5140.532330] LustreError: 164886:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5143.796110] LustreError: 164870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760302 with bad export cookie 2117938471558031589 [ 5143.800724] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5143.807767] LustreError: 164870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5158.370510] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5158.897734] Lustre: server umount lustre-MDT0001 complete [ 5172.089527] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 5174.995567] Key type lgssc unregistered [ 5175.270855] LNet: 169381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5175.287840] LNetError: 169381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5175.304643] LNet: Removed LNI 192.168.203.106@tcp [ 5176.192500] Key type .llcrypt unregistered [ 5176.194563] Key type ._llcrypt unregistered [ 5204.131710] Key type ._llcrypt registered [ 5204.137522] Key type .llcrypt registered [ 5204.236972] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5205.245173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5205.269341] alg: No test for adler32 (adler32-zlib) [ 5206.277748] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 5206.457986] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 5208.095165] Key type lgssc registered [ 5209.007685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5215.508928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5215.525616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5216.937660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5220.568446] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5226.468408] LustreError: 170795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5226.481569] LustreError: 170795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5228.800994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5229.122267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5233.380230] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5240.434782] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5245.817803] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5252.506737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5252.996310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5259.488982] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5261.219079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5268.783259] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5276.661521] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5276.669335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5276.677525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5276.706841] Lustre: Skipped 1 previous similar message [ 5277.643844] Lustre: server umount lustre-OST0000 complete [ 5281.760895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5281.766387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5281.775119] Lustre: Skipped 1 previous similar message [ 5281.786141] Lustre: Skipped 1 previous similar message [ 5286.880610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5286.892671] Lustre: Skipped 2 previous similar messages [ 5287.483330] Lustre: server umount lustre-MDT0000 complete [ 5290.774135] LustreError: 170776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760449 with bad export cookie 18277163538904776523 [ 5290.783155] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5290.793309] LustreError: 170776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5291.141076] Lustre: server umount lustre-MDT0001 complete [ 5301.681647] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 5304.148212] Key type lgssc unregistered [ 5304.473179] LNet: 174329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5304.495459] LNetError: 174329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5304.507780] LNet: Removed LNI 192.168.203.106@tcp [ 5305.371543] Key type .llcrypt unregistered [ 5305.386540] Key type ._llcrypt unregistered [ 5340.838307] Key type ._llcrypt registered [ 5340.840565] Key type .llcrypt registered [ 5340.962695] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5342.060142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5342.193033] alg: No test for adler32 (adler32-zlib) [ 5343.297284] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 5343.635229] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 5345.359372] Key type lgssc registered [ 5346.506283] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5354.241115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5354.262272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5355.855161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5361.231829] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5365.234352] LustreError: 176034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5365.251794] LustreError: 176034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5370.261263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5370.337679] LustreError: 176035:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5370.726817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5377.192564] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5387.154246] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5393.233520] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5400.567916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5400.938932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5406.898404] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5415.391459] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5425.527354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5435.874680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5435.879772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5435.886403] LustreError: Skipped 1 previous similar message [ 5435.888707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5435.917441] Lustre: Skipped 1 previous similar message [ 5438.978316] Lustre: server umount lustre-OST0000 complete [ 5446.113699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5446.119944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5446.124173] Lustre: Skipped 1 previous similar message [ 5446.149315] Lustre: Skipped 2 previous similar messages [ 5451.232821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5451.242518] Lustre: Skipped 1 previous similar message [ 5456.353024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5456.361024] Lustre: Skipped 1 previous similar message [ 5457.377413] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5457.583644] Lustre: server umount lustre-MDT0000 complete [ 5461.475319] LustreError: 176029:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5461.496342] LustreError: 176029:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5462.023365] LustreError: 176015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760621 with bad export cookie 269232433048351162 [ 5462.029780] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5462.038953] LustreError: 176015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5462.472525] Lustre: server umount lustre-MDT0001 complete [ 5474.619884] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 5477.778906] Key type lgssc unregistered [ 5478.101598] LNet: 179598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5478.110318] LNetError: 179598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5479.148160] LNet: Removed LNI 192.168.203.106@tcp [ 5480.236577] Key type .llcrypt unregistered [ 5480.244083] Key type ._llcrypt unregistered [ 5509.369529] Key type ._llcrypt registered [ 5509.372219] Key type .llcrypt registered [ 5509.464397] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5510.722277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5510.906500] alg: No test for adler32 (adler32-zlib) [ 5511.975874] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 5512.249702] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 5513.911799] Key type lgssc registered [ 5515.237972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5530.127130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5530.198781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5531.957584] Lustre: *** cfs_fail_loc=135, val=0*** [ 5532.217903] LustreError: 181139:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5532.230294] LustreError: 181139:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.106@tcp: cfg command failed: rc = -2 [ 5532.246882] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5532.246882] [ 5532.266100] LustreError: MGC192.168.203.106@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5532.298410] LustreError: 181100:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5532.318018] LustreError: 181100:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5532.340710] LustreError: 181100:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5532.349665] LustreError: 181100:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5532.484105] Lustre: server umount lustre-MDT0000 complete [ 5532.488639] LustreError: 181100:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5541.138268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.570747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5546.377793] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5551.084087] LustreError: 181512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5551.100074] LustreError: 181512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5555.834337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5556.116197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5561.051996] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5569.057265] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5575.556122] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5581.601494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5581.603139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5581.625690] Lustre: Skipped 1 previous similar message [ 5581.643749] Lustre: Skipped 1 previous similar message [ 5583.732673] Lustre: server umount lustre-MDT0000 complete [ 5586.918425] LustreError: 182511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5586.941451] LustreError: 182511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5587.513326] LustreError: 181492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760746 with bad export cookie 3821050334869812744 [ 5587.521850] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5588.028515] Lustre: server umount lustre-MDT0001 complete [ 5599.481794] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 5602.631291] Key type lgssc unregistered [ 5602.957775] LNet: 183835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5602.967463] LNetError: 183835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5602.999187] LNet: Removed LNI 192.168.203.106@tcp [ 5603.734071] Key type .llcrypt unregistered [ 5603.755079] Key type ._llcrypt unregistered [ 5625.679294] Key type ._llcrypt registered [ 5625.681838] Key type .llcrypt registered [ 5625.770750] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 5626.158372] Key type .llcrypt unregistered [ 5626.163185] Key type ._llcrypt unregistered [ 5645.091712] Key type ._llcrypt registered [ 5645.095171] Key type .llcrypt registered [ 5645.174551] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5646.467927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5646.511327] alg: No test for adler32 (adler32-zlib) [ 5647.535905] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 5647.778626] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 5649.431173] Key type lgssc registered [ 5650.343086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5657.425480] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5657.473620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5672.993633] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5678.111318] LustreError: 185973:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.106@tcp: failed processing log, type 4: rc = -110 [ 5704.800455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5712.537052] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5721.135977] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5728.811413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5729.145284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5729.241697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 5734.377807] LustreError: 185998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5734.396171] LustreError: 185998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5735.451496] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5739.494873] LustreError: 187009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5739.520996] LustreError: 187009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5744.619584] LustreError: 187009:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5744.666493] LustreError: 187009:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5746.938398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5747.590753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5752.611829] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5760.239803] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5766.296319] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5769.299724] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 5780.463184] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5780.481747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5780.504117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5783.520984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5783.533565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5783.541473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5783.572809] Lustre: Skipped 1 previous similar message [ 5785.734779] Lustre: server umount lustre-OST0000 complete [ 5793.761470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5793.765244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5793.786930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5793.797162] Lustre: Skipped 1 previous similar message [ 5793.812103] Lustre: Skipped 1 previous similar message [ 5796.207240] Lustre: server umount lustre-MDT0000 complete [ 5798.886427] LustreError: 187015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5798.918928] LustreError: 187015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5800.636518] LustreError: 188638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786760959 with bad export cookie 16783416420559695042 [ 5800.638917] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5800.653168] LustreError: 188638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5801.197282] Lustre: server umount lustre-MDT0001 complete [ 5814.300787] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 5817.101209] Key type lgssc unregistered [ 5817.452665] LNet: 189947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5817.467834] LNetError: 189947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5817.500328] LNet: Removed LNI 192.168.203.106@tcp [ 5818.546306] Key type .llcrypt unregistered [ 5818.547790] Key type ._llcrypt unregistered [ 5843.839985] Key type ._llcrypt registered [ 5843.842292] Key type .llcrypt registered [ 5843.928545] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 5845.426405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5845.470040] alg: No test for adler32 (adler32-zlib) [ 5846.643208] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 5847.036963] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 5848.735609] Key type lgssc registered [ 5850.383842] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5860.653471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5860.707950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5862.372061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5868.714158] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5872.611154] LustreError: 191360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5872.632641] LustreError: 191360:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5877.636041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5877.732919] LustreError: 191359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5877.958036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5882.723628] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5890.545875] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5897.487545] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5905.922780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5906.502801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5907.542635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 5914.419638] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5923.181828] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5928.966944] Lustre: Failing over lustre-MDT0000 [ 5929.271368] Lustre: server umount lustre-MDT0000 complete [ 5929.440666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5929.447400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5929.457144] LustreError: 191359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5933.031320] LustreError: 191360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5933.031901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5933.066726] LustreError: 191360:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5933.092445] Lustre: Skipped 1 previous similar message [ 5938.147054] LustreError: 191355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5938.158536] LustreError: 191355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5946.587327] LustreError: 191355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5946.609996] LustreError: 191355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5948.090042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5948.232204] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5948.395805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 5948.400655] Lustre: Skipped 2 previous similar messages [ 5948.624779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5948.737042] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5951.508023] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5953.346363] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 5954.044657] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5954.072608] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5954.149811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 5957.962267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 5967.290215] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 5982.193937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5982.210806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5982.232959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5983.168974] Lustre: server umount lustre-OST0000 complete [ 5989.867219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5989.874856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5989.879337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5989.889149] Lustre: Skipped 1 previous similar message [ 5989.913202] Lustre: Skipped 1 previous similar message [ 5992.416835] Lustre: server umount lustre-MDT0000 complete [ 5994.978717] LustreError: 191355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5995.003366] LustreError: 191355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5995.879299] LustreError: 191339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761154 with bad export cookie 15021163872108392482 [ 5995.887428] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5995.893511] LustreError: 191339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6005.394605] Lustre: server umount lustre-MDT0001 complete [ 6016.129564] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6018.538511] Key type lgssc unregistered [ 6018.872620] LNet: 196329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6018.888014] LNetError: 196329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6018.908503] LNet: Removed LNI 192.168.203.106@tcp [ 6019.846930] Key type .llcrypt unregistered [ 6019.850294] Key type ._llcrypt unregistered [ 6047.827622] Key type ._llcrypt registered [ 6047.830749] Key type .llcrypt registered [ 6047.922389] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6049.658744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6049.732780] alg: No test for adler32 (adler32-zlib) [ 6050.803780] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 6051.110536] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 6052.759277] Key type lgssc registered [ 6053.776263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6061.410325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6061.444545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6062.881195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6067.516872] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6072.289877] LustreError: 197742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6072.301619] LustreError: 197742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6076.778494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6077.432575] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6082.972970] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6092.726953] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6099.500807] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6106.369653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6106.738783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6107.769413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6112.748720] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6119.940573] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6122.832412] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6127.173260] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6147.162385] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6153.198281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6153.201200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6153.208208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6153.238683] Lustre: Skipped 1 previous similar message [ 6158.065567] Lustre: server umount lustre-OST0000 complete [ 6163.426213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6163.439671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6163.445293] Lustre: Skipped 1 previous similar message [ 6163.465472] Lustre: Skipped 2 previous similar messages [ 6168.544867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6168.556576] Lustre: Skipped 1 previous similar message [ 6169.065192] Lustre: server umount lustre-MDT0000 complete [ 6173.305552] LustreError: 197724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761332 with bad export cookie 4707872934396351980 [ 6173.313579] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6173.317433] LustreError: 197724:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6173.676570] Lustre: server umount lustre-MDT0001 complete [ 6184.840603] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6187.555792] Key type lgssc unregistered [ 6187.788628] LNet: 201609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6187.794063] LNetError: 201609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6187.818923] LNet: Removed LNI 192.168.203.106@tcp [ 6188.640451] Key type .llcrypt unregistered [ 6188.642450] Key type ._llcrypt unregistered [ 6208.659662] Key type ._llcrypt registered [ 6208.663671] Key type .llcrypt registered [ 6208.797073] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6210.024853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6210.037082] alg: No test for adler32 (adler32-zlib) [ 6211.129126] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 6211.378589] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 6213.023274] Key type lgssc registered [ 6213.999257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6221.954113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6221.993261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6223.439403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6227.745807] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6233.571575] LustreError: 202876:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.591240] LustreError: 202876:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6236.699961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6237.067027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6241.302808] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6248.593256] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6254.332578] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6261.036176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6261.385079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6267.472664] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6268.634761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6274.921856] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6281.387087] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6293.170227] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6299.962841] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6303.777544] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6313.964229] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6313.974540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6313.996434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6314.981416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6314.983390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6314.988759] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6315.016986] Lustre: Skipped 1 previous similar message [ 6317.338764] Lustre: server umount lustre-OST0000 complete [ 6324.192245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6324.196746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6324.206325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6326.890511] Lustre: server umount lustre-MDT0000 complete [ 6330.320121] LustreError: 202856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761489 with bad export cookie 931804463879861214 [ 6330.324210] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6330.340453] LustreError: 202856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6330.669294] Lustre: server umount lustre-MDT0001 complete [ 6341.293428] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6344.089568] Key type lgssc unregistered [ 6344.467589] LNet: 207658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6344.478325] LNetError: 207658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6344.496183] LNet: Removed LNI 192.168.203.106@tcp [ 6345.315909] Key type .llcrypt unregistered [ 6345.318116] Key type ._llcrypt unregistered [ 6368.864332] Key type ._llcrypt registered [ 6368.870597] Key type .llcrypt registered [ 6368.955658] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6369.989097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6370.058896] alg: No test for adler32 (adler32-zlib) [ 6371.062469] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 6371.379603] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 6373.057142] Key type lgssc registered [ 6374.038983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6382.529306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6382.556544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6384.214677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6389.302169] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6394.340688] LustreError: 209072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6394.377514] LustreError: 209072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6398.481846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6399.173873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6404.268492] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6413.834840] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6419.839265] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6426.547962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6427.053038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6429.132986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6434.368562] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6441.943543] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6459.872583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6459.874348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6459.879669] LustreError: Skipped 1 previous similar message [ 6459.884759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6459.895381] Lustre: Skipped 1 previous similar message [ 6463.413735] Lustre: server umount lustre-OST0000 complete [ 6470.126588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6470.141363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6470.141371] Lustre: Skipped 1 previous similar message [ 6470.158017] Lustre: Skipped 1 previous similar message [ 6473.883795] Lustre: server umount lustre-MDT0000 complete [ 6475.232452] LustreError: 210228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6475.261925] LustreError: 210228:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6478.148049] LustreError: 209052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761637 with bad export cookie 5491954909249115278 [ 6478.155361] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6478.164586] LustreError: 209052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6478.489900] Lustre: server umount lustre-MDT0001 complete [ 6492.093374] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6494.564162] Key type lgssc unregistered [ 6494.828715] LNet: 212759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6494.840716] LNetError: 212759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6494.881513] LNet: Removed LNI 192.168.203.106@tcp [ 6495.685553] Key type .llcrypt unregistered [ 6495.687291] Key type ._llcrypt unregistered [ 6517.028754] Key type ._llcrypt registered [ 6517.030932] Key type .llcrypt registered [ 6517.153604] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6518.292981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6518.351255] alg: No test for adler32 (adler32-zlib) [ 6519.483094] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 6519.777996] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 6521.471328] Key type lgssc registered [ 6522.438450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6530.572753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6530.586361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6532.104694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6536.581292] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6542.318061] LustreError: 214028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6542.344224] LustreError: 214028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6544.808854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6545.361152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6550.706973] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6558.762380] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6565.303297] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6571.639051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6572.042424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6577.149591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6578.692375] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6586.691848] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6600.673277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6600.677801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6600.689433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6605.803704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6605.809852] Lustre: Skipped 1 previous similar message [ 6606.420089] Lustre: server umount lustre-OST0000 complete [ 6610.914689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6610.936377] Lustre: Skipped 1 previous similar message [ 6610.944822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6610.952962] Lustre: Skipped 2 previous similar messages [ 6616.034098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6616.042545] Lustre: Skipped 1 previous similar message [ 6621.153046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6621.166416] Lustre: Skipped 1 previous similar message [ 6624.223307] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6624.377713] Lustre: server umount lustre-MDT0000 complete [ 6626.272245] LustreError: 214028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6626.288833] LustreError: 214028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6627.512926] LustreError: 214007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761786 with bad export cookie 5093037051355451409 [ 6627.518205] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6627.529169] LustreError: 214007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6627.778660] Lustre: server umount lustre-MDT0001 complete [ 6637.200423] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6639.348327] Key type lgssc unregistered [ 6639.647748] LNet: 217712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6639.662182] LNetError: 217712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6639.693865] LNet: Removed LNI 192.168.203.106@tcp [ 6640.371787] Key type .llcrypt unregistered [ 6640.379629] Key type ._llcrypt unregistered [ 6659.130337] Key type ._llcrypt registered [ 6659.136550] Key type .llcrypt registered [ 6659.253685] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6660.181602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6660.365717] alg: No test for adler32 (adler32-zlib) [ 6661.418357] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 6661.643217] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 6663.312888] Key type lgssc registered [ 6664.255844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6671.903891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6671.936890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6673.381249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6676.987458] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6683.622871] LustreError: 218978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6683.650270] LustreError: 218978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6684.777503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6685.140106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6689.120207] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6696.377499] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6701.744693] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6707.552639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6707.819708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6713.352377] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6718.058183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 6720.758512] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6728.551021] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6728.634267] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6728.877899] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6734.478359] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6741.624625] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6742.973291] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6755.292611] Lustre: Permanently deactivating lustre-OST0001 [ 6755.344327] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6762.606828] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 6769.041269] Lustre: Permanently reactivating lustre-OST0001 [ 6769.055485] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6769.067280] Lustre: Skipped 2 previous similar messages [ 6771.161333] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6771.188299] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6771.199680] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6771.218980] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6771.262248] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6781.410420] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6781.415493] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6781.435063] Lustre: Skipped 1 previous similar message [ 6781.443246] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6781.699644] Lustre: server umount lustre-OST0001 complete [ 6781.921708] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6781.927704] LustreError: 220798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6784.494355] LustreError: 220798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6784.516866] LustreError: 220798:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6784.992369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6785.006841] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6785.027803] Lustre: Skipped 1 previous similar message [ 6785.049386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6788.068192] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6788.077917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.096286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6789.620184] LustreError: 220798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6789.636485] LustreError: 220798:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6794.722990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6794.730748] LustreError: 222343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6794.739400] Lustre: Skipped 3 previous similar messages [ 6794.755307] LustreError: 222343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6799.330241] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6799.546234] Lustre: server umount lustre-OST0000 complete [ 6802.914539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6802.921919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6802.942486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6808.979086] Lustre: server umount lustre-MDT0000 complete [ 6810.086084] LustreError: 218977:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6810.099947] LustreError: 218977:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6812.295139] LustreError: 218960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786761971 with bad export cookie 17827793925884134619 [ 6812.295566] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6812.306923] LustreError: 218960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6812.787463] Lustre: server umount lustre-MDT0001 complete [ 6821.614543] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 6824.576367] Key type lgssc unregistered [ 6824.997990] LNet: 224023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6825.010232] LNetError: 224023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6825.029915] LNet: Removed LNI 192.168.203.106@tcp [ 6826.134235] Key type .llcrypt unregistered [ 6826.136556] Key type ._llcrypt unregistered [ 6866.034624] Key type ._llcrypt registered [ 6866.036633] Key type .llcrypt registered [ 6866.126709] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 6867.312976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6867.333803] alg: No test for adler32 (adler32-zlib) [ 6868.459677] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 6868.722565] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 6870.391272] Key type lgssc registered [ 6871.476575] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6878.574828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6878.611787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6879.860844] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6879.905080] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6880.100384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6883.605037] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6891.256041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6891.348201] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6891.370866] Lustre: 226224:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6891.564267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6895.561170] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6903.575795] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6909.112798] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6915.361855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6915.511045] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6915.707209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6921.275964] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 6921.888607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 6929.903625] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6934.100519] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6937.609460] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 6947.631204] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 6956.022348] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6971.326324] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6971.330477] Lustre: Skipped 1 previous similar message [ 6991.240494] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 6991.245966] Lustre: Skipped 2 previous similar messages [ 7024.710518] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7024.715501] Lustre: Skipped 3 previous similar messages [ 7097.432212] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7097.437691] Lustre: Skipped 8 previous similar messages [ 7106.016871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7106.032057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7106.054078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7106.530876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7106.544652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7106.560864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7111.666457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7111.671902] Lustre: Skipped 1 previous similar message [ 7112.064065] Lustre: server umount lustre-OST0000 complete [ 7116.773133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7116.775710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7116.798925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7121.849849] Lustre: server umount lustre-MDT0000 complete [ 7125.535679] LustreError: 226219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786762284 with bad export cookie 2341835850759190265 [ 7125.544534] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7125.548527] LustreError: 226219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7126.081416] Lustre: server umount lustre-MDT0001 complete [ 7135.810606] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 7138.340655] Key type lgssc unregistered [ 7138.641543] LNet: 231008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7138.653502] LNetError: 231008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7138.703312] LNet: Removed LNI 192.168.203.106@tcp [ 7139.518935] Key type .llcrypt unregistered [ 7139.521813] Key type ._llcrypt unregistered [ 7163.411144] Key type ._llcrypt registered [ 7163.416542] Key type .llcrypt registered [ 7163.543368] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7164.565440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7164.593867] alg: No test for adler32 (adler32-zlib) [ 7165.754666] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 7166.054886] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 7167.727221] Key type lgssc registered [ 7168.621604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7177.732278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7177.761914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7179.411440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7184.683988] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7188.973173] LustreError: 232424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7188.997426] LustreError: 232424:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7194.081435] LustreError: 232425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7194.906033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7195.253411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7199.575572] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7207.090675] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7212.724401] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7219.466616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7219.858103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7224.066597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7225.755170] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_default_debug -1 all [ 7233.583212] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7255.012769] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7255.014756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7255.034516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7255.048577] Lustre: Skipped 1 previous similar message [ 7258.445981] Lustre: server umount lustre-OST0000 complete [ 7265.250786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7265.280116] Lustre: Skipped 1 previous similar message [ 7265.289134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7265.301155] Lustre: Skipped 1 previous similar message [ 7268.986285] Lustre: server umount lustre-MDT0000 complete [ 7270.368580] LustreError: 232421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7270.408574] LustreError: 232421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7272.882527] LustreError: 232406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786762431 with bad export cookie 3760216396938581406 [ 7272.891225] LustreError: MGC192.168.203.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7272.895168] LustreError: 232406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7273.208448] Lustre: server umount lustre-MDT0001 complete [ 7283.939705] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 7286.961761] Key type lgssc unregistered [ 7287.221564] LNet: 236121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7287.234351] LNetError: 236121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7287.265036] LNet: Removed LNI 192.168.203.106@tcp [ 7288.174907] Key type .llcrypt unregistered [ 7288.176978] Key type ._llcrypt unregistered [ 7313.483490] Key type ._llcrypt registered [ 7313.486654] Key type .llcrypt registered [ 7313.641189] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7314.633279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7314.897265] alg: No test for adler32 (adler32-zlib) [ 7315.986162] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 7316.227393] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 7317.970737] Key type lgssc registered [ 7318.996485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7589.304190] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 7592.615938] Key type lgssc unregistered [ 7592.895633] LNet: 238149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7592.907696] LNetError: 238149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7592.919595] LNet: Removed LNI 192.168.203.106@tcp [ 7593.953423] Key type .llcrypt unregistered [ 7593.959337] Key type ._llcrypt unregistered [ 7623.421372] Key type ._llcrypt registered [ 7623.423739] Key type .llcrypt registered [ 7623.519084] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing set_hostid [ 7639.920340] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing load_modules_local [ 7641.305304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7641.339688] alg: No test for adler32 (adler32-zlib) [ 7642.357670] Lustre: Lustre: Build Version: 2.17.57_1_g2c756e6 [ 7642.598703] LNet: Added LNI 192.168.203.106@tcp [8/256/0/180] [ 7644.279176] Key type lgssc registered [ 7645.293830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7679.817963] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7386 sec ========== 23:00:37 (1786762837) [ 7681.877643] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:00:39 (1786762839) === [ 7685.791752] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:00:43 (1786762843) === [ 7728.416410] Lustre: DEBUG MARKER: oleg306-server.virtnet: executing unload_modules_local [ 7731.119961] Key type lgssc unregistered [ 7731.373514] LNet: 243402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7731.381483] LNetError: 243402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7731.396143] LNet: Removed LNI 192.168.203.106@tcp [ 7732.201503] Key type .llcrypt unregistered [ 7732.208068] Key type ._llcrypt unregistered