[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 505934065 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003251] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008558] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011143] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013038] SELinux: Initializing. [ 0.014063] *** VALIDATE selinux *** [ 0.021727] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025768] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027002] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029102] *** VALIDATE tmpfs *** [ 0.030447] *** VALIDATE proc *** [ 0.031205] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.033268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036030] Spectre V2 : User space: Vulnerable [ 0.037009] Speculative Store Bypass: Vulnerable [ 0.039743] debug: unmapping init [mem 0xffffffff93c59000-0xffffffff93c60fff] [ 0.041776] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042724] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043029] ... version: 2 [ 0.044015] ... bit width: 48 [ 0.045013] ... generic registers: 4 [ 0.046013] ... value mask: 0000ffffffffffff [ 0.047017] ... max period: 00007fffffffffff [ 0.048016] ... fixed-purpose events: 3 [ 0.049013] ... event mask: 000000070000000f [ 0.050309] rcu: Hierarchical SRCU implementation. [ 0.052695] smp: Bringing up secondary CPUs ... [ 0.053586] x86: Booting SMP configuration: [ 0.054024] .... node #0, CPUs: #1 #2 #3 [ 0.059012] smp: Brought up 1 node, 4 CPUs [ 0.061012] smpboot: Max logical packages: 1 [ 0.062018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137985] node 0 deferred pages initialised in 72ms [ 0.141098] devtmpfs: initialized [ 0.143318] x86/mm: Memory block size: 128MB [ 0.147186] gcov: version magic: 0x41383552 [ 0.150313] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151067] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155340] pinctrl core: initialized pinctrl subsystem [ 0.157179] [ 0.157578] ************************************************************* [ 0.159009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161015] ** ** [ 0.163016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166014] ** ** [ 0.168016] ** This means that this kernel is built to expose internal ** [ 0.170014] ** IOMMU data structures, which may compromise security on ** [ 0.171009] ** your system. ** [ 0.173022] ** ** [ 0.174013] ** If you see this message and you are not debugging the ** [ 0.175011] ** kernel, report this immediately to your vendor! ** [ 0.177013] ** ** [ 0.179011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182012] ************************************************************* [ 0.185192] NET: Registered protocol family 16 [ 0.187643] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195189] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199120] cpuidle: using governor menu [ 0.201972] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205497] PCI: Using configuration type 1 for base access [ 0.207130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217198] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223140] cryptd: max_cpu_qlen set to 1000 [ 0.226272] ACPI: Added _OSI(Module Device) [ 0.228012] ACPI: Added _OSI(Processor Device) [ 0.230014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.238992] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.246060] ACPI: Interpreter enabled [ 0.247055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.249011] ACPI: Using IOAPIC for interrupt routing [ 0.251108] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.255399] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.268278] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.270042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.273021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.276143] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.282858] acpiphp: Slot [2] registered [ 0.283132] acpiphp: Slot [5] registered [ 0.284121] acpiphp: Slot [6] registered [ 0.286134] acpiphp: Slot [7] registered [ 0.288157] acpiphp: Slot [8] registered [ 0.290210] acpiphp: Slot [9] registered [ 0.292170] acpiphp: Slot [10] registered [ 0.294169] acpiphp: Slot [3] registered [ 0.295100] acpiphp: Slot [4] registered [ 0.297117] acpiphp: Slot [11] registered [ 0.299112] acpiphp: Slot [12] registered [ 0.301118] acpiphp: Slot [13] registered [ 0.302105] acpiphp: Slot [14] registered [ 0.304207] acpiphp: Slot [15] registered [ 0.306186] acpiphp: Slot [16] registered [ 0.308110] acpiphp: Slot [17] registered [ 0.310190] acpiphp: Slot [18] registered [ 0.312223] acpiphp: Slot [19] registered [ 0.314137] acpiphp: Slot [20] registered [ 0.316111] acpiphp: Slot [21] registered [ 0.318137] acpiphp: Slot [22] registered [ 0.320202] acpiphp: Slot [23] registered [ 0.322183] acpiphp: Slot [24] registered [ 0.325158] acpiphp: Slot [25] registered [ 0.327103] acpiphp: Slot [26] registered [ 0.328110] acpiphp: Slot [27] registered [ 0.329100] acpiphp: Slot [28] registered [ 0.331123] acpiphp: Slot [29] registered [ 0.333153] acpiphp: Slot [30] registered [ 0.335104] acpiphp: Slot [31] registered [ 0.336059] PCI host bridge to bus 0000:00 [ 0.338021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.340028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.342022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.347032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.350043] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.353030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.355195] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.359241] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.363369] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.374027] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.378575] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.381020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.384020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.388021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.391807] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.394773] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.397039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.400836] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.407014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.421024] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.427013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.434053] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.441017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.451014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.481016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.493942] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.502015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.510017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.530014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.541113] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.548019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.553026] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.569041] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.580660] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.588019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.595018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.614026] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.626727] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.633023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.639019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.653024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.663011] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.671016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.678016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.698023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.714254] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.718463] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.723501] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.725525] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.728244] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.734801] iommu: Default domain type: Passthrough [ 0.736410] SCSI subsystem initialized [ 0.738176] ACPI: bus type USB registered [ 0.740168] usbcore: registered new interface driver usbfs [ 0.743145] usbcore: registered new interface driver hub [ 0.745089] usbcore: registered new device driver usb [ 0.747174] pps_core: LinuxPPS API ver. 1 registered [ 0.749014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.752090] PTP clock support registered [ 0.755112] EDAC MC: Ver: 3.0.0 [ 0.757179] PCI: Using ACPI for IRQ routing [ 0.758884] NetLabel: Initializing [ 0.759013] NetLabel: domain hash size = 128 [ 0.760016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.761147] NetLabel: unlabeled traffic allowed by default [ 0.764207] vgaarb: loaded [ 0.768342] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.772016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.782211] clocksource: Switched to clocksource kvm-clock [ 0.897116] VFS: Disk quotas dquot_6.6.0 [ 0.899283] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.902931] *** VALIDATE ramfs *** [ 0.904734] *** VALIDATE hugetlbfs *** [ 0.907055] pnp: PnP ACPI init [ 0.910581] pnp: PnP ACPI: found 6 devices [ 0.928385] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.932218] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.935709] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.939221] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.942532] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.945505] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.949431] NET: Registered protocol family 2 [ 0.952939] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.959287] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.963918] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.969726] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.973597] TCP: Hash tables configured (established 65536 bind 65536) [ 0.976681] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.980162] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.983252] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.986526] NET: Registered protocol family 1 [ 0.990662] RPC: Registered named UNIX socket transport module. [ 0.993132] RPC: Registered udp transport module. [ 0.995024] RPC: Registered tcp transport module. [ 0.996866] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.999255] NET: Registered protocol family 44 [ 1.000984] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.003042] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.005199] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.007551] PCI: CLS 0 bytes, default 64 [ 1.009302] Unpacking initramfs... [ 2.437996] debug: unmapping init [mem 0xffff93acfcc54000-0xffff93acfffbffff] [ 2.442433] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.444972] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.447975] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.952420] Initialise system trusted keyrings [ 2.954258] Key type blacklist registered [ 2.956558] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.966990] zbud: loaded [ 2.970782] *** VALIDATE nfs *** [ 2.972274] *** VALIDATE nfs4 *** [ 2.974204] pstore: using deflate compression [ 2.978252] Platform Keyring initialized [ 3.096278] NET: Registered protocol family 38 [ 3.098354] Key type asymmetric registered [ 3.099621] Asymmetric key parser 'x509' registered [ 3.101855] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.108091] io scheduler mq-deadline registered [ 3.110096] io scheduler kyber registered [ 3.111917] io scheduler bfq registered [ 3.114547] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.117722] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.120730] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.123752] ACPI: Power Button [PWRF] [ 3.130872] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.140617] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.155263] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.163146] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.190447] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.219858] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.248903] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.255299] Non-volatile memory driver v1.3 [ 3.257907] Linux agpgart interface v0.103 [ 3.306520] virtio_blk virtio1: [vda] 149896 512-byte logical blocks (76.7 MB/73.2 MiB) [ 3.310814] vda: detected capacity change from 0 to 76746752 [ 3.330651] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.333484] vdb: detected capacity change from 0 to 1073741824 [ 3.351106] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.354408] vdc: detected capacity change from 0 to 2621440000 [ 3.372605] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.376350] vdd: detected capacity change from 0 to 2621440000 [ 3.392469] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.396042] vde: detected capacity change from 0 to 4294967296 [ 3.415189] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.418412] vdf: detected capacity change from 0 to 4294967296 [ 3.431443] libphy: Fixed MDIO Bus: probed [ 3.439230] usbcore: registered new interface driver usbserial_generic [ 3.442068] usbserial: USB Serial support registered for generic [ 3.444671] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.449397] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.454148] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.456926] mousedev: PS/2 mouse device common for all mice [ 3.461868] rtc_cmos 00:05: RTC can wake from S4 [ 3.464145] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.473802] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.473860] rtc_cmos 00:05: registered as rtc0 [ 3.481924] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.484038] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.491422] intel_pstate: CPU model not supported [ 3.494745] hid: raw HID events driver (C) Jiri Kosina [ 3.497147] usbcore: registered new interface driver usbhid [ 3.499275] usbhid: USB HID core driver [ 3.500949] drop_monitor: Initializing network drop monitor service [ 3.503458] Initializing XFRM netlink socket [ 3.505700] NET: Registered protocol family 10 [ 3.509304] Segment Routing with IPv6 [ 3.510359] NET: Registered protocol family 17 [ 3.512091] mpls_gso: MPLS GSO support [ 3.517378] RAS: Correctable Errors collector initialized. [ 3.519697] AVX version of gcm_enc/dec engaged. [ 3.521577] AES CTR mode by8 optimization enabled [ 3.601389] sched_clock: Marking stable (3601344978, 0)->(4547837207, -946492229) [ 3.606711] registered taskstats version 1 [ 3.609545] Loading compiled-in X.509 certificates [ 3.612748] zswap: loaded using pool lzo/zbud [ 3.644150] Key type big_key registered [ 3.657707] Key type encrypted registered [ 3.659655] ima: No TPM chip found, activating TPM-bypass! [ 3.662323] ima: Allocated hash algorithm: sha1 [ 3.664752] ima: No architecture policies found [ 3.667495] evm: Initialising EVM extended attributes: [ 3.669875] evm: security.selinux [ 3.672526] evm: security.ima [ 3.673778] evm: security.capability [ 3.675226] evm: HMAC attrs: 0x1 [ 3.677493] rtc_cmos 00:05: setting system clock to 2026-09-03 08:02:02 UTC (1788422522) [ 3.685764] debug: unmapping init [mem 0xffffffff94c03000-0xffffffff94dfffff] [ 3.689580] debug: unmapping init [mem 0xffffffff93982000-0xffffffff93c58fff] [ 3.700081] Write protecting the kernel read-only data: 28672k [ 3.703552] debug: unmapping init [mem 0xffffffff92003000-0xffffffff921fffff] [ 3.706562] debug: unmapping init [mem 0xffffffff92914000-0xffffffff929fffff] [ 3.741373] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.751135] systemd[1]: Detected virtualization kvm. [ 3.752934] systemd[1]: Detected architecture x86-64. [ 3.755079] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.782711] systemd[1]: No hostname configured. [ 3.784847] systemd[1]: Set hostname to . [ 3.787082] random: systemd: uninitialized urandom read (16 bytes read) [ 3.790030] systemd[1]: Initializing machine ID from random generator. [ 3.853854] random: ln: uninitialized urandom read (6 bytes read) [ 3.951603] random: systemd: uninitialized urandom read (16 bytes read) [ 3.955400] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.963134] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.968373] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.685699] device-mapper: uevent: version 1.0.3 [ 4.688159] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.451936] virtio_net virtio0 ens2: renamed from eth0 [ 5.463765] scsi host0: ata_piix [ 5.483946] random: fast init done [ 5.489576] scsi host1: ata_piix [ 5.491803] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.495625] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.907713] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.180169] random: crng init done [ 10.181819] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.719916] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.998883] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.274253] SELinux: Disabled at runtime. [ 12.339147] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.348889] systemd[1]: Detected virtualization kvm. [ 12.350869] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.868255] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.872648] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.878026] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.884472] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.889121] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.898394] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.914925] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Created slic[ 12.990724] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS e User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.427912] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.785322] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.816780] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.889791] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.916272] EDAC sbridge: Ver: 1.1.2 [ 16.050464] Key type dns_resolver registered [ 16.385326] NFS: Registering the id_resolver key type [ 16.387285] Key type id_resolver registered [ 16.388969] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. Starting Network Manager... [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg437-server login: [ 52.860932] libcfs: loading out-of-tree module taints kernel. [ 52.906475] Key type ._llcrypt registered [ 52.909088] Key type .llcrypt registered [ 53.061116] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 71.460519] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 73.302419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 73.320105] alg: No test for adler32 (adler32-zlib) [ 74.775277] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 75.365994] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 77.135854] Key type lgssc registered [ 79.315288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 101.743458] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 148.531333] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 161.896468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 161.917767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 163.252778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 163.310914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 163.387675] Lustre: lustre-MDT0000: new disk, initializing [ 163.472415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.490258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 167.583565] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.073783] hrtimer: interrupt took 2119306 ns [ 181.165984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.242305] Lustre: 6512:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 181.280119] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 181.283548] Lustre: Skipped 1 previous similar message [ 181.342256] Lustre: lustre-MDT0001: new disk, initializing [ 181.392901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 181.410324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 181.425858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 185.913134] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 191.256522] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 204.710647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 205.222816] Lustre: lustre-OST0000: new disk, initializing [ 205.226499] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 205.234288] Lustre: 8447:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 205.305801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.758194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.774266] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 207.848814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 212.596216] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.310807] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 228.462308] Lustre: lustre-OST0001: new disk, initializing [ 228.469257] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 228.474250] Lustre: 9522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 228.583495] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 235.354343] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.697730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 237.705291] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 237.765487] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 246.705449] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 253.697219] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 259.490914] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing check_logdir /tmp/testlogs/ [ 265.149297] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing yml_node [ 269.867931] Lustre: DEBUG MARKER: Client: 2.17.58.7 [ 272.459219] Lustre: DEBUG MARKER: MDS: 2.17.58.7 [ 274.680422] Lustre: DEBUG MARKER: OSS: 2.17.58.7 [ 276.261663] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 04:06:32 EDT 2026 [ 290.800918] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 292.526792] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 309.728955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.731132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 309.735022] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 309.751501] Lustre: Skipped 2 previous similar messages [ 309.794417] Lustre: Skipped 3 previous similar messages [ 314.432513] Lustre: server umount lustre-MDT0000 complete [ 318.264063] LustreError: 10338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788422837 with bad export cookie 3104965968034427606 [ 318.268057] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 318.290897] LustreError: 10338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 319.968735] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.969077] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.970464] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 319.998912] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 320.044249] Lustre: Skipped 2 previous similar messages [ 324.004899] LustreError: 6519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 324.005837] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 324.059462] Lustre: Skipped 2 previous similar messages [ 324.651923] Lustre: server umount lustre-MDT0001 complete [ 329.003660] Lustre: server umount lustre-OST0000 complete [ 332.565933] Lustre: server umount lustre-OST0001 complete [ 339.264599] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 348.393238] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 387.953973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 388.120141] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 388.145660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 388.276692] Lustre: lustre-MDT0000: new disk, initializing [ 388.360530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 388.400777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 394.229743] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 405.289615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.524919] Lustre: 15074:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 405.686293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 405.697992] Lustre: Skipped 1 previous similar message [ 405.841437] Lustre: lustre-MDT0001: new disk, initializing [ 405.936332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 406.004026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 406.015964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 411.635807] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 424.850743] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 432.902789] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 439.612715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 440.108307] Lustre: lustre-OST0000: new disk, initializing [ 440.114531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 440.122578] Lustre: 17042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 440.189315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 441.990155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 441.996702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 442.154760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 447.208730] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 458.695356] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 464.127150] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 464.354929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 469.275440] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 469.557810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 473.057436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 473.058990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.067710] LustreError: Skipped 1 previous similar message [ 473.071935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 473.094155] Lustre: Skipped 1 previous similar message [ 473.113509] Lustre: Skipped 1 previous similar message [ 477.895301] Lustre: server umount lustre-OST0000 complete [ 482.785741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 482.798054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.847344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 488.110785] Lustre: server umount lustre-MDT0000 complete [ 488.418623] LustreError: 15086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 488.440639] LustreError: 15086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 491.829473] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423010 with bad export cookie 3104965968034429293 [ 491.848386] LustreError: 15068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 491.860362] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 492.277821] Lustre: server umount lustre-MDT0001 complete [ 495.873553] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 04:10:12 (1788423012) [ 503.161185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.720607] LustreError: 19312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.801942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.901136] LustreError: 19313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 509.481250] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 519.150819] LustreError: 19312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 519.169800] LustreError: 19312:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 519.426391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 524.832610] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 532.573854] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 538.984432] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 545.728159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 553.878963] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 561.984915] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 573.921493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 573.934575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 573.942123] LustreError: Skipped 1 previous similar message [ 573.973121] Lustre: Skipped 1 previous similar message [ 573.979272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 573.984575] Lustre: Skipped 2 previous similar messages [ 578.347390] Lustre: server umount lustre-OST0000 complete [ 585.197532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 585.240408] Lustre: Skipped 2 previous similar messages [ 588.925096] Lustre: server umount lustre-MDT0000 complete [ 590.308451] LustreError: 19308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.352070] LustreError: 19308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 593.963365] LustreError: 19294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423112 with bad export cookie 3104965968034430217 [ 593.981919] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 593.983683] LustreError: 19294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 594.549683] Lustre: server umount lustre-MDT0001 complete [ 608.411308] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 612.102248] Key type lgssc unregistered [ 612.578216] LNet: 22842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.584755] LNetError: 22842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 613.607716] LNet: Removed LNI 192.168.204.137@tcp [ 615.010174] Key type .llcrypt unregistered [ 615.012550] Key type ._llcrypt unregistered [ 639.068764] Key type ._llcrypt registered [ 639.074327] Key type .llcrypt registered [ 639.255836] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 640.211979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 640.323101] alg: No test for adler32 (adler32-zlib) [ 641.395535] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 641.686618] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 643.408189] Key type lgssc registered [ 644.674193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 653.664554] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 653.698217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 655.219977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 659.924892] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 665.574267] LustreError: 24255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 665.600848] LustreError: 24255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 668.419534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 668.899097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 673.060161] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 680.385748] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 686.037378] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 692.054042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 692.517239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 698.447472] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 700.725443] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 705.425935] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 721.386629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 721.391195] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 721.422495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 721.441229] Lustre: Skipped 1 previous similar message [ 722.576624] Lustre: server umount lustre-OST0000 complete [ 726.498527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 726.511290] Lustre: Skipped 1 previous similar message [ 726.520338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 726.528620] Lustre: Skipped 1 previous similar message [ 731.616952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 731.623974] Lustre: Skipped 1 previous similar message [ 732.570820] Lustre: server umount lustre-MDT0000 complete [ 736.023179] LustreError: 24234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423254 with bad export cookie 12680553656357898424 [ 736.027055] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 736.036909] LustreError: 24234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 736.475536] Lustre: server umount lustre-MDT0001 complete [ 746.452168] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 749.174319] Key type lgssc unregistered [ 749.458217] LNet: 28230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.468615] LNetError: 28230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.499182] LNet: Removed LNI 192.168.204.137@tcp [ 750.266947] Key type .llcrypt unregistered [ 750.270542] Key type ._llcrypt unregistered [ 769.694474] Key type ._llcrypt registered [ 769.696288] Key type .llcrypt registered [ 769.836660] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 770.959437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 771.037765] alg: No test for adler32 (adler32-zlib) [ 772.131463] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 772.344568] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 774.023189] Key type lgssc registered [ 775.261540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 783.266429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 783.284560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 784.755074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 789.475688] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 795.104682] LustreError: 29620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 795.137775] LustreError: 29620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 797.923321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 798.388924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 802.975365] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 814.250896] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 822.497549] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 835.968213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 836.269650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 842.267214] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 850.078531] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 854.723953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 862.485261] Lustre: server umount lustre-OST0000 complete [ 869.856388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 869.866608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 869.891832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 870.376412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 870.401655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 870.411925] Lustre: Skipped 1 previous similar message [ 875.489488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 875.504573] Lustre: Skipped 1 previous similar message [ 876.594546] Lustre: server umount lustre-MDT0000 complete [ 880.547267] Lustre: 28932:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788423383/real 1788423383] req@0000000073d3be98 x1875297740815616/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788423399 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 880.551922] LustreError: 32460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423399 with bad export cookie 12330668606793139193 [ 880.562176] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 880.582249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.596893] LustreError: 32460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 881.275751] Lustre: server umount lustre-MDT0001 complete [ 892.235692] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 895.009488] Key type lgssc unregistered [ 895.452492] LNet: 33498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.462616] LNetError: 33498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.480342] LNet: Removed LNI 192.168.204.137@tcp [ 896.608469] Key type .llcrypt unregistered [ 896.610874] Key type ._llcrypt unregistered [ 923.511354] Key type ._llcrypt registered [ 923.513483] Key type .llcrypt registered [ 923.601538] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 924.455118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 924.467156] alg: No test for adler32 (adler32-zlib) [ 925.668437] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 925.942794] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 927.615212] Key type lgssc registered [ 929.093553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 938.846775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 938.879294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 940.558233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 944.758483] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 950.752667] LustreError: 34908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 950.769822] LustreError: 34908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 953.066343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 953.517161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 957.562879] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 965.110180] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 970.537919] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 978.537930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 979.154456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 986.312321] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 986.386468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 995.345423] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1007.073687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1007.084577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1007.118934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1009.123734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1009.130760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1009.138690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1009.160918] Lustre: Skipped 1 previous similar message [ 1013.363784] Lustre: server umount lustre-OST0000 complete [ 1019.363320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.366752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1019.381960] Lustre: Skipped 1 previous similar message [ 1024.184310] Lustre: server umount lustre-MDT0000 complete [ 1024.481352] LustreError: 35636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1024.491847] LustreError: 35636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1028.275287] LustreError: 37693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423547 with bad export cookie 9204229044686495859 [ 1028.280379] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1028.296168] LustreError: 37693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 1028.786551] Lustre: server umount lustre-MDT0001 complete [ 1041.390859] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 1044.143184] Key type lgssc unregistered [ 1044.424739] LNet: 38444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.434711] LNetError: 38444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.456781] LNet: Removed LNI 192.168.204.137@tcp [ 1045.281926] Key type .llcrypt unregistered [ 1045.288729] Key type ._llcrypt unregistered [ 1069.942672] Key type ._llcrypt registered [ 1069.946912] Key type .llcrypt registered [ 1070.046544] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 1071.474804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1071.567915] alg: No test for adler32 (adler32-zlib) [ 1072.724636] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 1072.987643] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 1074.727159] Key type lgssc registered [ 1076.019586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1083.901442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1083.919904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.559513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1089.698863] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1096.162162] LustreError: 39855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1096.174172] LustreError: 39855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1098.952243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1099.346436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1104.219332] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1111.509471] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1116.466305] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1123.061475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1123.587629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.982450] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1130.758707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1135.960965] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1143.582678] Lustre: lustre-OST0000: Not available for connect from 192.168.204.37@tcp (stopping) [ 1145.317953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1145.332508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1145.370964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1146.356308] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.373853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1146.374836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1146.391989] Lustre: Skipped 1 previous similar message [ 1148.668829] LustreError: 41676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1148.866813] Lustre: server umount lustre-OST0000 complete [ 1170.912414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1170.920591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.938618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1178.080430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1178.085181] Lustre: Skipped 5 previous similar messages [ 1183.201394] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1183.421607] Lustre: server umount lustre-MDT0000 complete [ 1186.729671] LustreError: 39837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423705 with bad export cookie 16623515995940040774 [ 1186.737467] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1186.737706] LustreError: 39837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1201.119959] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1201.379188] Lustre: server umount lustre-MDT0001 complete [ 1212.069622] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 1215.845450] Key type lgssc unregistered [ 1216.114489] LNet: 43565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.129575] LNetError: 43565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1216.146916] LNet: Removed LNI 192.168.204.137@tcp [ 1217.149354] Key type .llcrypt unregistered [ 1217.151089] Key type ._llcrypt unregistered [ 1245.726327] Key type ._llcrypt registered [ 1245.727934] Key type .llcrypt registered [ 1245.823778] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 1247.038369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1247.136477] alg: No test for adler32 (adler32-zlib) [ 1248.344326] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 1248.803102] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 1250.575219] Key type lgssc registered [ 1251.767779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1260.627965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1260.641349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1262.003090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1266.505879] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1271.278620] LustreError: 44975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1271.292598] LustreError: 44975:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1274.972313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1275.405709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1280.361977] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1288.538914] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1294.708397] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1302.350681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1302.835441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1306.948338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1309.021802] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1315.917931] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1325.561653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1325.574557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1325.577911] Lustre: Skipped 1 previous similar message [ 1325.589377] Lustre: Skipped 2 previous similar messages [ 1327.900815] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.37@tcp (stopping) [ 1330.662734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1330.677092] Lustre: Skipped 2 previous similar messages [ 1331.249626] Lustre: server umount lustre-MDT0000 complete [ 1332.988490] LustreError: 44971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.512995] LustreError: 44955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788423854 with bad export cookie 10334953825120169499 [ 1335.530121] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1335.538992] LustreError: 44955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1335.777248] LustreError: 44970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.778786] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.839682] Lustre: Skipped 1 previous similar message [ 1335.857921] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1338.123642] LustreError: 44970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.898940] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1340.914076] Lustre: Skipped 1 previous similar message [ 1342.163617] Lustre: server umount lustre-MDT0001 complete [ 1358.155642] Lustre: server umount lustre-OST0000 complete [ 1374.755992] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 1377.158174] Key type lgssc unregistered [ 1377.478354] LNet: 48836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.486907] LNetError: 48836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.504074] LNet: Removed LNI 192.168.204.137@tcp [ 1378.266538] Key type .llcrypt unregistered [ 1378.269105] Key type ._llcrypt unregistered [ 1399.447791] Key type ._llcrypt registered [ 1399.451619] Key type .llcrypt registered [ 1399.570516] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 1400.519888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1400.557091] alg: No test for adler32 (adler32-zlib) [ 1401.618866] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 1401.823420] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 1403.487261] Key type lgssc registered [ 1404.694404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1412.635900] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1412.656924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1428.191380] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1433.311350] LustreError: 50200:0:(mgc_request.c:317:do_config_log_add()) mgc192.168.204.137@tcp: failed processing log, type 4: rc = -110 [ 1460.063210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1467.676894] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1479.259163] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1647.341634] Lustre: server umount lustre-OST0000 complete [ 1667.574673] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 1670.598848] Key type lgssc unregistered [ 1670.958412] LNet: 51877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1670.972938] LNetError: 51877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1670.992881] LNet: Removed LNI 192.168.204.137@tcp [ 1671.961383] Key type .llcrypt unregistered [ 1671.963959] Key type ._llcrypt unregistered [ 1697.410618] Key type ._llcrypt registered [ 1697.413627] Key type .llcrypt registered [ 1697.513446] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 1698.352803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1698.589600] alg: No test for adler32 (adler32-zlib) [ 1699.745912] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 1699.970828] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 1701.719962] Key type lgssc registered [ 1703.238720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1713.814367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1713.827369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1715.543643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1720.517248] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1725.924294] LustreError: 53287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1725.936264] LustreError: 53287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1729.870606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.350621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1735.530927] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1743.184896] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1749.214414] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1757.178824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1757.775136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1760.826787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1765.178679] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1773.800075] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1780.708207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1780.712413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.720191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1780.750204] Lustre: Skipped 1 previous similar message [ 1783.681436] Lustre: server umount lustre-OST0000 complete [ 1790.945774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1790.976687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1790.981739] Lustre: Skipped 2 previous similar messages [ 1793.845632] Lustre: server umount lustre-MDT0000 complete [ 1796.090406] LustreError: 54525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1796.110576] LustreError: 54525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1798.190996] LustreError: 53267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788424317 with bad export cookie 8987756088294547346 [ 1798.197723] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1798.208801] LustreError: 53267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1798.816713] Lustre: server umount lustre-MDT0001 complete [ 1810.086882] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 1813.964247] Key type lgssc unregistered [ 1814.451617] LNet: 56818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.458819] LNetError: 56818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.482753] LNet: Removed LNI 192.168.204.137@tcp [ 1816.068035] Key type .llcrypt unregistered [ 1816.070427] Key type ._llcrypt unregistered [ 1842.561585] Key type ._llcrypt registered [ 1842.564107] Key type .llcrypt registered [ 1842.675346] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 1844.130347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1844.195265] alg: No test for adler32 (adler32-zlib) [ 1845.233861] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 1845.606487] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 1847.407508] Key type lgssc registered [ 1848.578441] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1856.836755] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1856.873181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1858.008360] Lustre: 58211:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1873.570986] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1878.687355] LustreError: 58199:0:(mgc_request.c:317:do_config_log_add()) mgc192.168.204.137@tcp: failed processing log, type 4: rc = -110 [ 1883.105239] Lustre: 58213:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1889.055836] Lustre: 58199:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 1899.385234] Lustre: 58211:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1899.552591] Lustre: 58215:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 1904.608483] Lustre: 58213:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1909.663235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1909.733043] Lustre: 58211:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1909.855283] LustreError: 58215:0:(mgc_request.c:718:do_requeue()) failed processing log: -110 [ 1916.861864] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1919.970642] Lustre: 58211:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 1920.001927] Lustre: 58211:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 1 previous similar message [ 1925.751078] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1930.435082] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1936.714551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1937.241867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1937.345522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1942.514886] LustreError: 58229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1942.532063] LustreError: 58229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1944.568332] LustreError: 59244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1944.616389] LustreError: 59244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1944.652894] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1949.676026] LustreError: 59244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1949.704102] LustreError: 59244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1954.790741] LustreError: 59243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1954.838718] LustreError: 59243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1957.394570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1957.801231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1963.570201] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 1972.435978] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1979.014646] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1983.461735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1983.465761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.466941] LustreError: Skipped 1 previous similar message [ 1983.467924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1983.483907] Lustre: Skipped 1 previous similar message [ 1987.357403] Lustre: server umount lustre-OST0000 complete [ 2014.204505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2014.207153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2014.235381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.235390] Lustre: Skipped 1 previous similar message [ 2019.024782] Lustre: server umount lustre-MDT0000 complete [ 2019.297292] LustreError: 59249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2019.344970] LustreError: 59249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2025.173050] LustreError: 58207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788424543 with bad export cookie 7635878033709973303 [ 2025.177632] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2025.194287] LustreError: 58207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2031.717600] Lustre: server umount lustre-MDT0001 complete [ 2048.231651] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 2053.926383] Key type lgssc unregistered [ 2054.599079] LNet: 61959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2054.623944] LNetError: 61959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2054.653068] LNet: Removed LNI 192.168.204.137@tcp [ 2056.575386] Key type .llcrypt unregistered [ 2056.579268] Key type ._llcrypt unregistered [ 2085.915511] Key type ._llcrypt registered [ 2085.918463] Key type .llcrypt registered [ 2086.115154] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 2086.829805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2086.858715] alg: No test for adler32 (adler32-zlib) [ 2087.894600] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 2088.199035] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 2089.992407] Key type lgssc registered [ 2091.399125] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2100.052182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2100.085567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2101.754430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.264913] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2111.973812] LustreError: 63378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2112.011359] LustreError: 63378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2117.089677] LustreError: 63377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2120.429880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2120.883989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2126.624384] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2138.680096] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2145.765305] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2152.908800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2153.215387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2159.727654] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2166.872408] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2172.666402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2186.209112] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2186.218334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2186.237602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2188.257947] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2188.267873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2188.272044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.298336] Lustre: Skipped 1 previous similar message [ 2190.090583] Lustre: server umount lustre-OST0000 complete [ 2197.984848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2197.995107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2198.024477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2201.048252] Lustre: server umount lustre-MDT0000 complete [ 2203.624122] LustreError: 64339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2203.640890] LustreError: 64339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2205.529942] LustreError: 63359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788424724 with bad export cookie 4842166641669636834 [ 2205.538157] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2205.546996] LustreError: 63359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2206.093234] Lustre: server umount lustre-MDT0001 complete [ 2221.041762] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 2223.912687] Key type lgssc unregistered [ 2224.435714] LNet: 66938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2224.451401] LNetError: 66938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2224.473286] LNet: Removed LNI 192.168.204.137@tcp [ 2225.276787] Key type .llcrypt unregistered [ 2225.279058] Key type ._llcrypt unregistered [ 2252.940220] Key type ._llcrypt registered [ 2252.944731] Key type .llcrypt registered [ 2253.115412] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 2254.081834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2254.229877] alg: No test for adler32 (adler32-zlib) [ 2255.322951] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 2255.595943] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 2257.279183] Key type lgssc registered [ 2258.517317] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2267.948436] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 04:39:44 (1788424784) [ 2275.383833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2275.414849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2276.996215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2281.809476] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2286.645861] LustreError: 68642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2286.687568] LustreError: 68642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2290.969744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2291.462383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2296.025744] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2304.179038] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2310.052245] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2316.516442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2316.892444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.053475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2324.082446] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2331.686871] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2341.308089] Lustre: Failing over lustre-MDT0000 [ 2341.551517] Lustre: server umount lustre-MDT0000 complete [ 2341.860225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.867219] LustreError: 68639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2341.871821] Lustre: Skipped 1 previous similar message [ 2341.892174] LustreError: 68639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2344.735777] LustreError: 70728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2346.980312] LustreError: 68639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2347.002481] LustreError: 68639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2349.558224] LustreError: 68622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788424868 with bad export cookie 4765457086374863027 [ 2349.567207] LustreError: mgc192.168.204.137@tcp: operation mgs_target_reg to node 0@lo failed: rc = -107 [ 2349.568939] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2349.570748] Lustre: 68634:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2349.570938] Lustre: 71689:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2349.602982] LustreError: Skipped 1 previous similar message [ 2350.115725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2350.574364] Lustre: *** cfs_fail_loc=135, val=0*** [ 2350.693554] LustreError: 71735:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2350.702198] LustreError: 71735:0:(obd_config.c:2100:class_config_llog_handler()) mgc192.168.204.137@tcp: cfg command failed: rc = -2 [ 2350.714330] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2350.714330] [ 2350.719680] LustreError: mgc192.168.204.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2350.725870] LustreError: 71723:0:(tgt_mount.c:2005:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2350.731168] LustreError: 71723:0:(tgt_mount.c:2799:server_fill_super()) Unable to start targets: -2 [ 2350.736186] LustreError: 71723:0:(tgt_mount.c:2249:server_put_super()) no obd lustre-MDT0000 [ 2350.739858] LustreError: 71723:0:(tgt_mount.c:130:server_deregister_mount()) lustre-MDT0000 not registered [ 2350.756930] Lustre: server umount lustre-MDT0000 complete [ 2350.759504] LustreError: 71723:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2352.097217] LustreError: 69371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.124195] LustreError: 69371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2357.238938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2357.321767] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2357.566203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2357.630910] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2360.064533] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2362.109723] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2362.873126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2362.897665] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2362.947701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2372.584185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2372.594973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.612248] Lustre: Skipped 1 previous similar message [ 2372.616923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2375.494929] Lustre: server umount lustre-OST0000 complete [ 2383.328445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2383.338303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2383.350653] Lustre: Skipped 2 previous similar messages [ 2383.361894] Lustre: Skipped 3 previous similar messages [ 2386.008430] Lustre: server umount lustre-MDT0000 complete [ 2388.453460] LustreError: 68642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.473982] LustreError: 68642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2389.630464] LustreError: 68622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788424908 with bad export cookie 4765457086374864259 [ 2389.632271] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2389.643036] LustreError: 68622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2390.020728] Lustre: server umount lustre-MDT0001 complete [ 2400.711814] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 2404.762161] Key type lgssc unregistered [ 2405.289229] LNet: 73513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2405.300617] LNetError: 73513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2405.335820] LNet: Removed LNI 192.168.204.137@tcp [ 2406.443765] Key type .llcrypt unregistered [ 2406.449323] Key type ._llcrypt unregistered [ 2432.500336] Key type ._llcrypt registered [ 2432.502019] Key type .llcrypt registered [ 2432.613802] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 2433.651367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2433.700964] alg: No test for adler32 (adler32-zlib) [ 2434.761713] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 2435.084412] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 2436.751269] Key type lgssc registered [ 2437.994565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2446.853625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2446.870204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2448.389753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2452.673782] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2458.591914] LustreError: 74923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.622879] LustreError: 74923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2461.695659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2462.042454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2466.532667] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2474.790489] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2480.158550] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2486.950486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2487.292647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2493.431875] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2493.438120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2501.208460] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2511.173637] Lustre: Failing over lustre-MDT0000 [ 2511.418708] Lustre: server umount lustre-MDT0000 complete [ 2511.847324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2511.852983] LustreError: 74919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2511.866444] Lustre: Skipped 1 previous similar message [ 2511.887884] LustreError: 74919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2514.202495] LustreError: 77339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2516.978074] LustreError: 74924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2517.030308] LustreError: 74924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2519.025665] LustreError: mgc192.168.204.137@tcp: operation ldlm_enqueue to node 0@lo failed: rc = -107 [ 2519.029529] LustreError: 74904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425037 with bad export cookie 745455713600391181 [ 2519.039722] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2519.046159] Lustre: 77887:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 2519.059611] Lustre: 74915:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -107 [ 2520.734313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2521.089824] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2521.200887] LustreError: 78015:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2521.206550] LustreError: 78015:0:(obd_config.c:2100:class_config_llog_handler()) mgc192.168.204.137@tcp: cfg command failed: rc = -9 [ 2521.214265] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2521.214265] [ 2521.225659] LustreError: mgc192.168.204.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2521.249841] LustreError: 78004:0:(tgt_mount.c:2005:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2521.275338] LustreError: 78004:0:(tgt_mount.c:2799:server_fill_super()) Unable to start targets: -9 [ 2521.293425] LustreError: 78004:0:(tgt_mount.c:2249:server_put_super()) no obd lustre-MDT0000 [ 2521.307338] LustreError: 78004:0:(tgt_mount.c:130:server_deregister_mount()) lustre-MDT0000 not registered [ 2521.344052] Lustre: server umount lustre-MDT0000 complete [ 2521.354468] LustreError: 78004:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2522.083755] LustreError: 74923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.095685] LustreError: 74923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2527.580582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.681977] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2527.904030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2527.974120] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2529.532498] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2532.586313] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2533.368318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2533.405647] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2533.496017] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2542.560926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2542.587651] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2542.613232] Lustre: Skipped 1 previous similar message [ 2542.622515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2543.591131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2543.596273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2546.536589] Lustre: server umount lustre-OST0000 complete [ 2553.829565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2553.837300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.838679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2553.865739] Lustre: Skipped 2 previous similar messages [ 2556.603777] Lustre: server umount lustre-MDT0000 complete [ 2558.946247] LustreError: 74924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2558.963290] LustreError: 74924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2559.997224] LustreError: 78841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425078 with bad export cookie 745455713600392427 [ 2560.003063] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2560.010238] LustreError: 78841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2560.431306] Lustre: server umount lustre-MDT0001 complete [ 2571.880893] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 2574.648441] Key type lgssc unregistered [ 2574.982582] LNet: 79794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2574.990850] LNetError: 79794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2575.005214] LNet: Removed LNI 192.168.204.137@tcp [ 2575.830666] Key type .llcrypt unregistered [ 2575.844394] Key type ._llcrypt unregistered [ 2588.372591] Key type ._llcrypt registered [ 2588.375702] Key type .llcrypt registered [ 2588.528972] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 2588.882673] Key type .llcrypt unregistered [ 2588.889625] Key type ._llcrypt unregistered [ 2613.401124] Key type ._llcrypt registered [ 2613.404904] Key type .llcrypt registered [ 2613.598539] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 2614.976727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2615.021231] alg: No test for adler32 (adler32-zlib) [ 2616.153968] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 2616.429740] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 2618.167172] Key type lgssc registered [ 2619.478433] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2628.122654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2628.159208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2629.732489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.302204] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2639.846365] LustreError: 81693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2639.865064] LustreError: 81693:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2643.714867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2644.262334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2648.671612] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2656.289735] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2661.903187] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2668.317405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2668.777820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2675.015786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2676.066240] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2683.672957] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2695.664616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2695.664850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2695.678623] LustreError: Skipped 1 previous similar message [ 2695.683719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2695.687748] Lustre: Skipped 1 previous similar message [ 2695.724114] Lustre: Skipped 1 previous similar message [ 2698.961556] Lustre: server umount lustre-OST0000 complete [ 2705.889080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2705.893461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2705.907624] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2705.924164] Lustre: Skipped 1 previous similar message [ 2705.940469] Lustre: Skipped 1 previous similar message [ 2711.008743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2716.129190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2716.133302] Lustre: Skipped 2 previous similar messages [ 2717.151168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2717.337295] Lustre: server umount lustre-MDT0000 complete [ 2721.254163] LustreError: 81688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2721.272416] LustreError: 81688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2721.286990] LustreError: 84272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425240 with bad export cookie 12924310569022204118 [ 2721.294575] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2721.295550] LustreError: 84272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2721.864044] Lustre: server umount lustre-MDT0001 complete [ 2732.472095] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 2735.496844] Key type lgssc unregistered [ 2735.912573] LNet: 85225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2735.922527] LNetError: 85225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2735.936753] LNet: Removed LNI 192.168.204.137@tcp [ 2736.800744] Key type .llcrypt unregistered [ 2736.803844] Key type ._llcrypt unregistered [ 2759.573583] Key type ._llcrypt registered [ 2759.576125] Key type .llcrypt registered [ 2759.664387] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 2760.566035] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2760.699357] alg: No test for adler32 (adler32-zlib) [ 2761.837708] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 2762.122256] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 2763.815247] Key type lgssc registered [ 2764.894688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2772.578245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2772.605569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.072936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2778.709289] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2783.211227] LustreError: 86643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2783.239851] LustreError: 86643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2787.549663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2788.006771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2793.167912] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2800.847515] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2807.031594] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2814.215584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2814.612967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.744391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2820.461897] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2827.560374] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2838.506313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2838.506982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2838.539179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2842.280861] Lustre: server umount lustre-OST0000 complete [ 2848.741771] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2848.746424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2848.760602] Lustre: Skipped 2 previous similar messages [ 2848.766626] Lustre: Skipped 1 previous similar message [ 2852.374963] Lustre: server umount lustre-MDT0000 complete [ 2853.865772] LustreError: 86638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.886115] LustreError: 86638:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2856.279394] LustreError: 88486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425375 with bad export cookie 8337865988975719694 [ 2856.291442] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2856.302158] LustreError: 88486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2856.701268] Lustre: server umount lustre-MDT0001 complete [ 2867.965944] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 2870.874547] Key type lgssc unregistered [ 2871.135767] LNet: 90176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2871.144635] LNetError: 90176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2871.162079] LNet: Removed LNI 192.168.204.137@tcp [ 2871.863601] Key type .llcrypt unregistered [ 2871.866956] Key type ._llcrypt unregistered [ 2895.389923] Key type ._llcrypt registered [ 2895.393604] Key type .llcrypt registered [ 2895.632267] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 2897.263725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2897.285370] alg: No test for adler32 (adler32-zlib) [ 2898.383898] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 2898.639187] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 2900.335208] Key type lgssc registered [ 2901.619627] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2909.599341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2909.625862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2911.020260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2915.117250] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2920.420567] LustreError: 91591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2920.445097] LustreError: 91591:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2923.427822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2923.773235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2928.188836] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2935.116825] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2940.704713] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2946.909705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2947.371935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2953.416118] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 2955.573079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 2961.277290] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2971.106404] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2971.114301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2971.118647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2971.127026] Lustre: Skipped 1 previous similar message [ 2973.132585] Lustre: server umount lustre-OST0000 complete [ 2980.320388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2980.330770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2980.344854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2980.350538] Lustre: Skipped 1 previous similar message [ 2981.361543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2981.362136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2981.383435] Lustre: Skipped 1 previous similar message [ 2982.680925] Lustre: server umount lustre-MDT0000 complete [ 2985.880868] LustreError: 91572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425504 with bad export cookie 8379312163043968672 [ 2985.887933] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2985.894830] LustreError: 91572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2986.208859] Lustre: server umount lustre-MDT0001 complete [ 2996.661730] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 2999.594558] Key type lgssc unregistered [ 2999.911632] LNet: 95123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2999.918611] LNetError: 95123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2999.943762] LNet: Removed LNI 192.168.204.137@tcp [ 3000.774000] Key type .llcrypt unregistered [ 3000.777218] Key type ._llcrypt unregistered [ 3025.604339] Key type ._llcrypt registered [ 3025.608363] Key type .llcrypt registered [ 3025.749952] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 3026.687817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3026.741322] alg: No test for adler32 (adler32-zlib) [ 3027.853921] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 3028.102901] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 3029.775166] Key type lgssc registered [ 3030.980450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3038.845345] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3038.859592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3039.939278] Lustre: 96519:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3055.455488] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3060.578853] LustreError: 96506:0:(mgc_request.c:317:do_config_log_add()) mgc192.168.204.137@tcp: failed processing log, type 4: rc = -110 [ 3065.315389] Lustre: 96521:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3070.879340] Lustre: 96506:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 3081.119623] Lustre: 96523:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 3081.144762] Lustre: 96519:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3086.304358] Lustre: 96521:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3090.407261] Lustre: 96519:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3091.360141] LustreError: 96523:0:(mgc_request.c:718:do_requeue()) failed processing log: -110 [ 3091.423249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3099.060297] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3100.646769] Lustre: 96519:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3100.661090] Lustre: 96519:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 1 previous similar message [ 3101.599858] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3106.961854] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3121.119793] Lustre: 96521:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3121.131856] Lustre: 96521:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 3 previous similar messages [ 3138.017606] Lustre: 96523:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 3138.211495] Lustre: server umount lustre-OST0000 complete [ 3148.222924] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 04:54:24 (1788425664) [ 3155.789344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3156.501627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3161.262693] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3161.571377] LustreError: 98159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3161.581686] LustreError: 98159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3166.693724] LustreError: 98158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3170.173658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3170.633477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3176.113096] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3183.058471] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3188.621733] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3196.091382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3196.616026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3201.717635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3204.400291] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3213.762403] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3231.713271] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3231.724457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3231.732414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3231.750205] Lustre: Skipped 1 previous similar message [ 3234.511940] Lustre: server umount lustre-OST0000 complete [ 3241.955401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3241.969108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3241.972358] Lustre: Skipped 2 previous similar messages [ 3244.776666] Lustre: server umount lustre-MDT0000 complete [ 3247.072112] LustreError: 98155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3247.090032] LustreError: 98155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3248.010295] LustreError: 100897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425766 with bad export cookie 6156932024144086741 [ 3248.015067] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3248.022636] LustreError: 100897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3248.296795] Lustre: server umount lustre-MDT0001 complete [ 3257.968778] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 3260.408174] Key type lgssc unregistered [ 3260.781841] LNet: 101850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3260.792100] LNetError: 101850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3260.813335] LNet: Removed LNI 192.168.204.137@tcp [ 3261.808553] Key type .llcrypt unregistered [ 3261.811404] Key type ._llcrypt unregistered [ 3304.142780] Key type ._llcrypt registered [ 3304.145682] Key type .llcrypt registered [ 3304.241828] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 3305.528805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3305.576057] alg: No test for adler32 (adler32-zlib) [ 3306.663981] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 3306.873064] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 3308.505068] Key type lgssc registered [ 3309.412143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3316.273627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3316.298557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3317.724548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3322.175649] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3327.976079] LustreError: 104033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3327.991841] LustreError: 104033:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3329.838352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3330.378257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3334.077234] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3341.178371] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3346.169386] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3352.179490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3352.569425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3358.419768] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3362.807035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3365.346716] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3376.098712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3376.108553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3376.127125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3378.159414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3378.164588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3378.165718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3378.172327] Lustre: Skipped 1 previous similar message [ 3381.098288] Lustre: server umount lustre-OST0000 complete [ 3386.852345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3386.861655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3386.874874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3391.381198] Lustre: server umount lustre-MDT0000 complete [ 3393.505713] LustreError: 104029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.523976] LustreError: 104029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3395.052037] LustreError: 104015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788425913 with bad export cookie 8218471998921589647 [ 3395.054187] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3395.065037] LustreError: 104015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3395.522913] Lustre: server umount lustre-MDT0001 complete [ 3408.367842] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 3411.169551] Key type lgssc unregistered [ 3411.569696] LNet: 107566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3411.574090] LNetError: 107566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3411.589089] LNet: Removed LNI 192.168.204.137@tcp [ 3412.254996] Key type .llcrypt unregistered [ 3412.258095] Key type ._llcrypt unregistered [ 3430.240936] Key type ._llcrypt registered [ 3430.245390] Key type .llcrypt registered [ 3430.413230] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 3432.076372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3432.249708] alg: No test for adler32 (adler32-zlib) [ 3433.427179] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 3433.759691] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 3435.484057] Key type lgssc registered [ 3437.261567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3447.175917] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3447.204916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3448.336128] Lustre: 108822:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3463.839409] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3468.959478] LustreError: 108808:0:(mgc_request.c:317:do_config_log_add()) mgc192.168.204.137@tcp: failed processing log, type 4: rc = -110 [ 3473.384924] Lustre: 108823:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3479.327382] Lustre: 108808:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 3489.594563] Lustre: 108822:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3491.168741] Lustre: 108825:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 3494.880191] Lustre: 108823:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3499.871206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3500.013144] Lustre: 108822:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3501.407939] LustreError: 108825:0:(mgc_request.c:718:do_requeue()) failed processing log: -110 [ 3506.839428] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3510.240033] Lustre: 108822:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 3510.249200] Lustre: 108822:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 1 previous similar message [ 3514.867385] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3521.890820] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3522.791666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.256540] LustreError: mgc192.168.204.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3523.273857] LustreError: 109840:0:(tgt_mount.c:2005:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3523.303716] LustreError: 109840:0:(tgt_mount.c:2799:server_fill_super()) Unable to start targets: -2 [ 3523.317527] LustreError: 109840:0:(tgt_mount.c:2249:server_put_super()) no obd lustre-MDT0000 [ 3523.460457] Lustre: server umount lustre-MDT0000 complete [ 3523.468160] LustreError: 109840:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3540.450122] Lustre: 108144:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788426043/real 1788426043] req@00000000d59187e8 x1875300531049472/t0(0) o400->mgc192.168.204.137@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788426059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3540.472599] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3561.088455] Lustre: server umount lustre-OST0000 complete [ 3570.622707] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 3579.763846] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 3614.485495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3614.755237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3614.808553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3614.877828] Lustre: lustre-MDT0000: new disk, initializing [ 3614.940303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3614.952343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3619.576708] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3629.186100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3629.287533] Lustre: 112648:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3629.458191] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3629.467239] Lustre: Skipped 1 previous similar message [ 3629.585442] Lustre: lustre-MDT0001: new disk, initializing [ 3629.691449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3629.748172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3629.771920] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3634.161918] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3643.828472] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3650.070269] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3656.566764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3656.870529] Lustre: lustre-OST0000: new disk, initializing [ 3656.875637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3656.882830] Lustre: 114615:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3657.023477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3663.351147] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3664.099861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3664.143964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3664.337872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3672.626580] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3678.208626] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3678.531657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3683.328070] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3683.537647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3689.954965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3689.959662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3689.969606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3689.998050] Lustre: Skipped 1 previous similar message [ 3691.981082] Lustre: server umount lustre-OST0000 complete [ 3700.196920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3700.199924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3700.209939] Lustre: Skipped 1 previous similar message [ 3700.222347] Lustre: Skipped 2 previous similar messages [ 3703.582879] Lustre: server umount lustre-MDT0000 complete [ 3707.990887] LustreError: 112639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788426226 with bad export cookie 4301689934857038303 [ 3707.999674] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3708.005531] LustreError: 112639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3708.547919] Lustre: server umount lustre-MDT0001 complete [ 3719.302068] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 05:03:55 (1788426235) [ 3723.936230] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3744.249323] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 3755.388055] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 3800.677307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3801.077754] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3801.126972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3801.352358] Lustre: lustre-MDT0000: new disk, initializing [ 3801.468506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.532614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3807.252463] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3820.157444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.262955] Lustre: 119518:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3820.363921] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3820.372844] Lustre: Skipped 1 previous similar message [ 3820.474892] Lustre: lustre-MDT0001: new disk, initializing [ 3820.601717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3820.650697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3820.663413] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3826.238245] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3836.454887] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3842.719429] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3849.323512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3849.703994] Lustre: lustre-OST0000: new disk, initializing [ 3849.716199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3849.726135] Lustre: 121487:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3849.809802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3857.217924] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3859.565769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3859.576926] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3859.690354] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3868.663623] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3874.904669] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3875.289364] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3880.724801] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3880.923791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3885.548613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3885.562236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3885.580245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3889.726522] Lustre: server umount lustre-OST0000 complete [ 3895.776945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3895.811245] Lustre: Skipped 1 previous similar message [ 3895.831636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3895.842948] Lustre: Skipped 2 previous similar messages [ 3900.897409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3900.918208] Lustre: Skipped 1 previous similar message [ 3909.090359] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3909.308702] Lustre: server umount lustre-MDT0000 complete [ 3911.141462] LustreError: 119529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3911.165400] LustreError: 119529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3913.013994] LustreError: 121486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788426431 with bad export cookie 4301689934857039227 [ 3913.017136] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3913.031705] LustreError: 121486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3913.443932] Lustre: server umount lustre-MDT0001 complete [ 3922.162513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3922.488320] LustreError: 123704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3922.565933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3926.776718] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3928.052728] LustreError: 123705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3933.159095] LustreError: 123704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3936.159487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3941.559630] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3948.076384] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3954.064236] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3962.403128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3970.292816] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 3977.128115] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3986.771171] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 3992.035339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3992.040722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3992.051800] Lustre: Skipped 2 previous similar messages [ 3992.065580] Lustre: Skipped 2 previous similar messages [ 3996.375168] Lustre: server umount lustre-OST0000 complete [ 4002.274285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4005.913259] Lustre: server umount lustre-MDT0000 complete [ 4007.397287] LustreError: 126197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4007.405648] LustreError: 126197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4009.624529] LustreError: 123684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788426528 with bad export cookie 4301689934857040144 [ 4009.641480] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4009.644465] LustreError: 123684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4010.060739] Lustre: server umount lustre-MDT0001 complete [ 4021.377441] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 4023.713262] Key type lgssc unregistered [ 4023.976631] LNet: 127330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4023.997157] LNetError: 127330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4024.024843] LNet: Removed LNI 192.168.204.137@tcp [ 4024.887182] Key type .llcrypt unregistered [ 4024.889598] Key type ._llcrypt unregistered [ 4043.807627] Key type ._llcrypt registered [ 4043.811604] Key type .llcrypt registered [ 4043.994400] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 4064.399376] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 4065.568334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4066.206812] alg: No test for adler32 (adler32-zlib) [ 4067.395290] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 4067.588557] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 4069.231440] Key type lgssc registered [ 4070.535789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4115.720881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4115.785078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4117.079590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4117.144645] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4117.426898] Lustre: lustre-MDT0000: new disk, initializing [ 4117.630529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4117.660888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4124.035445] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4136.769932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4136.927516] Lustre: 130486:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4137.017675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4137.025537] Lustre: Skipped 1 previous similar message [ 4137.078800] Lustre: lustre-MDT0001: new disk, initializing [ 4137.207056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4137.286808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4137.302259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4143.621305] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4156.078283] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4165.643677] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4178.539182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4179.379250] Lustre: lustre-OST0000: new disk, initializing [ 4179.419995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4179.438507] Lustre: 132461:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4179.700032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4185.821566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4185.844285] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4186.043761] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4190.399518] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4203.079277] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4208.480414] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4208.750325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4214.643944] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4214.987556] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4221.922582] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4221.931658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4221.944632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4221.962955] Lustre: Skipped 1 previous similar message [ 4222.006632] Lustre: Skipped 1 previous similar message [ 4227.052494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4227.066870] Lustre: Skipped 1 previous similar message [ 4232.164878] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4232.181412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4232.199528] Lustre: Skipped 1 previous similar message [ 4232.615931] Lustre: server umount lustre-OST0000 complete [ 4239.840797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4239.858715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4239.892116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4242.417574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4246.580218] Lustre: server umount lustre-MDT0000 complete [ 4251.767583] LustreError: 132460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788426770 with bad export cookie 3636376298911943010 [ 4251.775245] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4251.784366] LustreError: 132460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4252.392096] Lustre: server umount lustre-MDT0001 complete [ 4264.048611] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 05:12:59 (1788426779) [ 4271.620264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4272.012850] LustreError: 134878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4272.032453] LustreError: 134878:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4272.105864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4277.243995] LustreError: 134879:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4277.483740] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4281.378504] LustreError: 134878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4286.227404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4286.503956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4291.939129] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4299.863383] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4304.785719] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4306.916859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4306.917165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4306.921313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4306.921322] Lustre: Skipped 2 previous similar messages [ 4306.973404] Lustre: Skipped 1 previous similar message [ 4318.176985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4318.189156] Lustre: Skipped 6 previous similar messages [ 4321.248766] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4321.487982] Lustre: server umount lustre-MDT0000 complete [ 4323.300642] LustreError: 135618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4326.805915] LustreError: 134860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788426845 with bad export cookie 3636376298911943934 [ 4326.812731] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4326.833261] LustreError: 134860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4327.724590] Lustre: server umount lustre-MDT0001 complete [ 4337.372427] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 05:14:13 (1788426853) [ 4343.966509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4344.108311] Lustre: 137307:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4349.409405] Lustre: 137310:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4354.530255] Lustre: 137307:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4359.648306] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4359.653787] Lustre: 137310:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4364.769476] LustreError: 137295:0:(mgc_request.c:317:do_config_log_add()) mgc192.168.204.137@tcp: failed processing log, type 4: rc = -110 [ 4364.783050] Lustre: 137307:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4375.009257] Lustre: 137307:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4375.023331] Lustre: 137307:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 1 previous similar message [ 4375.071819] Lustre: 137295:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 4385.380968] Lustre: 137312:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 4395.679317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4395.681303] LustreError: 137312:0:(mgc_request.c:718:do_requeue()) failed processing log: -110 [ 4395.999678] Lustre: 137307:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4396.019550] Lustre: 137307:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 3 previous similar messages [ 4402.485396] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4405.921621] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4411.473192] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4414.385268] Lustre: server umount lustre-OST0000 complete [ 4425.203613] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 05:15:40 (1788426940) [ 4433.175908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4434.234172] LustreError: 138751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4434.283399] LustreError: 138751:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4434.394425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4439.978847] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4444.650418] LustreError: 138751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4444.684478] LustreError: 138751:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4449.909698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4455.667828] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4464.817478] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4470.551307] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4476.944427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4477.272843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4477.280654] Lustre: Skipped 1 previous similar message [ 4483.444033] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4492.038861] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4506.081498] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4506.092298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4511.999712] Lustre: server umount lustre-MDT0000 complete [ 4515.628969] LustreError: 138732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788427034 with bad export cookie 3636376298911944690 [ 4515.641475] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4515.654660] LustreError: 138732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4515.977821] Lustre: server umount lustre-MDT0001 complete [ 4530.567273] Lustre: server umount lustre-OST0000 complete [ 4538.785846] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 05:17:34 (1788427054) [ 4545.193670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4545.814294] LustreError: 142435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4545.838504] LustreError: 142435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4545.895282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4550.117840] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4560.002233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4568.108944] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4578.243702] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4584.884466] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4593.629165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4601.614291] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4609.914444] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4615.700853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4616.759244] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4617.128925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4622.719171] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4623.081339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4631.009353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4631.012787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4631.041628] Lustre: Skipped 3 previous similar messages [ 4631.049521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4631.049691] Lustre: Skipped 6 previous similar messages [ 4632.501892] Lustre: server umount lustre-OST0000 complete [ 4643.785381] Lustre: server umount lustre-MDT0000 complete [ 4646.375629] LustreError: 142436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4646.458955] LustreError: 142436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4649.826789] LustreError: 142416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788427168 with bad export cookie 3636376298911945817 [ 4649.847056] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4649.851131] LustreError: 142416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4650.625487] Lustre: server umount lustre-MDT0001 complete [ 4666.557081] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 05:19:41 (1788427181) [ 4674.916125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4674.979286] Lustre: 146611:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4674.986485] Lustre: 146611:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 3 previous similar messages [ 4690.406185] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4695.519590] LustreError: 146598:0:(mgc_request.c:317:do_config_log_add()) mgc192.168.204.137@tcp: failed processing log, type 4: rc = -110 [ 4705.824175] Lustre: 146598:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 4716.255879] Lustre: 146615:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 4726.367358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4726.375936] Lustre: Skipped 2 previous similar messages [ 4726.504337] LustreError: 146615:0:(mgc_request.c:718:do_requeue()) failed processing log: -110 [ 4733.075313] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4741.097922] Lustre: 146610:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4741.117127] Lustre: 146610:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 13 previous similar messages [ 4741.906675] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4747.041550] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4750.221882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4750.972180] LustreError: 146628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4751.297767] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4757.185762] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4769.031481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4774.621271] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4786.024749] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4795.141052] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4802.768338] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4803.356119] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4809.399744] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4810.132335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4815.328796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4815.337463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4815.363623] Lustre: Skipped 2 previous similar messages [ 4815.375533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4815.382326] Lustre: Skipped 2 previous similar messages [ 4819.749448] Lustre: server umount lustre-MDT0000 complete [ 4824.100354] LustreError: 146606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788427342 with bad export cookie 3636376298911946972 [ 4824.107078] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4824.112150] LustreError: 146606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4824.652906] Lustre: server umount lustre-MDT0001 complete [ 4839.422053] Lustre: server umount lustre-OST0000 complete [ 4849.041346] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 05:22:44 (1788427364) [ 4857.676451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4873.184307] Lustre: 150821:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 4873.208642] Lustre: 150821:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 4 previous similar messages [ 4873.375764] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4878.496401] LustreError: 150808:0:(mgc_request.c:317:do_config_log_add()) mgc192.168.204.137@tcp: failed processing log, type 4: rc = -110 [ 4888.866647] Lustre: 150808:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 4899.487680] Lustre: 150825:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 4909.727485] LustreError: 150825:0:(mgc_request.c:718:do_requeue()) failed processing log: -110 [ 4916.707087] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4925.424710] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4930.207797] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4932.208264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4932.624255] LustreError: 150838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4932.650280] LustreError: 150838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 4932.780240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4937.562315] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4947.993078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4955.784120] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4967.729878] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4974.664289] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4982.841392] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4983.181878] Lustre: lustre-OST0001: new disk, initializing [ 4983.192057] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4983.209298] Lustre: 153693:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4983.451238] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4983.455816] Lustre: Skipped 5 previous similar messages [ 4991.034755] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 4993.285712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4993.296617] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4993.532412] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5001.788732] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5006.373890] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5006.619523] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5011.584137] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5011.760533] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5013.993371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5014.027449] Lustre: Skipped 2 previous similar messages [ 5014.043530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5014.055081] Lustre: Skipped 3 previous similar messages [ 5019.819602] Lustre: server umount lustre-OST0000 complete [ 5024.224286] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5029.600441] Lustre: server umount lustre-OST0001 complete [ 5039.601563] Lustre: server umount lustre-MDT0000 complete [ 5044.212758] LustreError: 150815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788427563 with bad export cookie 3636376298911948057 [ 5044.227803] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5044.675478] Lustre: server umount lustre-MDT0001 complete [ 5070.233862] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 05:26:26 (1788427586) [ 5072.074149] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5073.800343] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5075.976600] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 05:26:31 (1788427591) [ 5082.363043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5082.590988] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5082.647739] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5087.623343] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5098.561020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5098.696128] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5098.753419] Lustre: 157253:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5104.882222] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5113.670242] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5120.359178] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5127.830209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5128.030075] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5134.546106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5135.215314] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5144.065791] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5151.250597] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5151.491147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5157.431263] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5157.764438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5160.419292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5160.421344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5160.433777] LustreError: Skipped 2 previous similar messages [ 5160.479508] Lustre: Skipped 6 previous similar messages [ 5166.457521] Lustre: server umount lustre-OST0000 complete [ 5190.789904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5196.797228] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.811458] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5196.814758] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5198.231720] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5206.354154] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5212.091909] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5212.325233] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5218.193515] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5218.520441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5225.025703] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9dee983e2800.ost_server_uuid 50 [ 5226.774900] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9dee983e2800.ost_server_uuid in FULL state after 0 sec [ 5227.030401] LustreError: 157262:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.37@tcp) returned error from blocking AST (req@00000000979dc717 x1875301196144768 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000e3275e6f/0x327700fa3e4be71b lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.37@tcp remote: 0x81c5df811c4b9f64 expref: 5 pid: 157261 timeout: 5327 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5227.063470] LustreError: lustre-MDT0000: A client on nid 192.168.204.37@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5227.071058] LustreError: 157251:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.37@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000e3275e6f/0x327700fa3e4be71b lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.37@tcp remote: 0x81c5df811c4b9f64 expref: 6 pid: 157261 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5232.615272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5238.144606] Lustre: server umount lustre-OST0000 complete [ 5248.005164] LustreError: 157261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5248.037919] LustreError: 157261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 5248.167788] Lustre: server umount lustre-MDT0000 complete [ 5252.158501] LustreError: 157246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788427770 with bad export cookie 3636376298911949359 [ 5252.164860] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5252.171209] LustreError: 157246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5258.615071] Lustre: server umount lustre-MDT0001 complete [ 5269.743527] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5272.824275] Key type lgssc unregistered [ 5273.095591] LNet: 163370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5273.106402] LNetError: 163370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5273.131844] LNet: Removed LNI 192.168.204.137@tcp [ 5274.101163] Key type .llcrypt unregistered [ 5274.102790] Key type ._llcrypt unregistered [ 5299.236689] Key type ._llcrypt registered [ 5299.239846] Key type .llcrypt registered [ 5299.370941] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 5300.332167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5300.540533] alg: No test for adler32 (adler32-zlib) [ 5301.672702] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 5301.949038] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 5303.680555] Key type lgssc registered [ 5304.802931] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5313.897297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5313.934513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5315.498661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5320.417468] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5325.797061] LustreError: 164779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5325.829139] LustreError: 164779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5329.551122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5330.122921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5335.966950] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5343.681332] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5349.606520] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5356.537422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5356.929375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5361.078300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5364.059650] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5372.065921] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5380.717524] Lustre: Failing over lustre-MDT0000 [ 5381.089467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5381.094384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5381.117854] LustreError: 164780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5381.180527] Lustre: server umount lustre-MDT0000 complete [ 5384.677434] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5384.677766] LustreError: 164779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5384.709656] Lustre: Skipped 1 previous similar message [ 5384.730065] LustreError: 164779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5389.804180] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5389.805378] LustreError: 164779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5389.824388] Lustre: 167478:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5389.848035] LustreError: 164779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5393.492295] LustreError: 164761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.37@tcp arrived at 1788427912 with bad export cookie 6454981576865628032 [ 5393.509955] LustreError: 164761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5394.919045] LustreError: 164780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5394.921543] Lustre: 167480:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5394.932060] LustreError: 164780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5394.959291] Lustre: 167480:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 1 previous similar message [ 5400.035994] Lustre: 167481:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5401.105247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5401.712558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5401.776662] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5406.398450] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5407.204334] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5407.225955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5408.687047] Lustre: lustre-MDT0000: Denying connection for new client fe2a744f-9251-400c-8050-9869e727bda7 (at 192.168.204.37@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:08 [ 5417.451387] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5417.458471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5417.469738] Lustre: Skipped 1 previous similar message [ 5417.482200] Lustre: Skipped 1 previous similar message [ 5422.561660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5427.682292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5427.693563] Lustre: Skipped 2 previous similar messages [ 5431.263194] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5431.598756] Lustre: server umount lustre-OST0000 complete [ 5435.561989] LustreError: 168522:0:(ldlm_lib.c:3046:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5435.576555] Lustre: 167872:0:(ldlm_lib.c:2446:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5435.585512] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5435.610107] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5435.620403] LustreError: 167872:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000056109d30 x1875302489836672/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5435.638524] LustreError: 167872:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5435.648202] LustreError: 167872:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5435.658538] Lustre: lustre-MDT0000: Recovery over after 0:28, of 2 clients 0 recovered and 2 were evicted. [ 5436.032489] Lustre: server umount lustre-MDT0000 complete [ 5437.920448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5437.963551] LustreError: 164775:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5437.977712] LustreError: 164775:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5440.164371] LustreError: 167483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788427958 with bad export cookie 6454981576865628165 [ 5440.179840] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5440.183380] LustreError: 167483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5440.201142] LustreError: 164088:0:(import.c:751:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5442.755657] Lustre: server umount lustre-MDT0001 complete [ 5455.458738] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5458.970987] Key type lgssc unregistered [ 5459.406075] LNet: 169273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5459.415342] LNetError: 169273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5459.438585] LNet: Removed LNI 192.168.204.137@tcp [ 5460.554924] Key type .llcrypt unregistered [ 5460.558949] Key type ._llcrypt unregistered [ 5489.976945] Key type ._llcrypt registered [ 5489.982453] Key type .llcrypt registered [ 5490.172674] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 5491.488146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5491.713656] alg: No test for adler32 (adler32-zlib) [ 5493.318358] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 5493.607614] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 5495.343165] Key type lgssc registered [ 5496.820698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5507.110440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5507.152543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5508.877732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5515.725437] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5518.309896] LustreError: 170683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5518.344056] LustreError: 170683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5523.429254] LustreError: 170684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5528.554831] LustreError: 170683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5529.953042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5530.583461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5536.320305] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5543.297762] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5549.582976] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5557.649345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5558.125785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5565.857435] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5578.111446] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5578.603566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5588.975499] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5588.980608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5589.007469] Lustre: Skipped 1 previous similar message [ 5589.021965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5592.895137] Lustre: server umount lustre-OST0000 complete [ 5597.171215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5597.185901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5597.222606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5597.231132] Lustre: Skipped 1 previous similar message [ 5599.211331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5599.233927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.579566] Lustre: server umount lustre-MDT0000 complete [ 5604.322107] LustreError: 170680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5604.348476] LustreError: 170680:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5607.757108] LustreError: 170665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788428126 with bad export cookie 9111853984587408822 [ 5607.764732] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5608.106651] Lustre: server umount lustre-MDT0001 complete [ 5619.631452] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5622.382841] Key type lgssc unregistered [ 5622.721652] LNet: 174216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5622.730433] LNetError: 174216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5622.754276] LNet: Removed LNI 192.168.204.137@tcp [ 5623.622898] Key type .llcrypt unregistered [ 5623.625520] Key type ._llcrypt unregistered [ 5656.623945] Key type ._llcrypt registered [ 5656.629309] Key type .llcrypt registered [ 5656.736649] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 5657.848893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5657.892074] alg: No test for adler32 (adler32-zlib) [ 5659.094883] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 5659.466159] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 5661.215432] Key type lgssc registered [ 5662.627724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5672.892256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5672.921821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5674.550281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5680.739278] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5684.711577] LustreError: 175916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5684.747400] LustreError: 175916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5689.824825] LustreError: 175915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5691.418157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5691.805581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5696.134898] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5704.860718] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5710.816446] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5718.191348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5718.618422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5719.677293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5726.370410] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5735.774526] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5742.565837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5742.579551] Lustre: Skipped 1 previous similar message [ 5742.580487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5742.584832] LustreError: 177742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5742.719624] Lustre: server umount lustre-OST0000 complete [ 5746.313238] Lustre: server umount lustre-MDT0000 complete [ 5747.688221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5747.695230] LustreError: 177050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5747.706063] Lustre: Skipped 1 previous similar message [ 5747.718818] LustreError: 177050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5749.467171] LustreError: 175895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788428268 with bad export cookie 12953694754165462130 [ 5749.481857] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5749.491321] LustreError: 175895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5749.824969] Lustre: server umount lustre-MDT0001 complete [ 5760.095326] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5763.667673] Key type lgssc unregistered [ 5764.176691] LNet: 179453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5764.185406] LNetError: 179453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5764.198189] LNet: Removed LNI 192.168.204.137@tcp [ 5765.328047] Key type .llcrypt unregistered [ 5765.331562] Key type ._llcrypt unregistered [ 5793.555655] Key type ._llcrypt registered [ 5793.557878] Key type .llcrypt registered [ 5793.646665] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 5794.591324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5794.693621] alg: No test for adler32 (adler32-zlib) [ 5795.846796] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 5796.165526] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 5797.863140] Key type lgssc registered [ 5799.388206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5814.895928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5814.938330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5816.528776] Lustre: *** cfs_fail_loc=135, val=0*** [ 5816.730285] LustreError: 180974:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5816.737686] LustreError: 180974:0:(obd_config.c:2100:class_config_llog_handler()) mgc192.168.204.137@tcp: cfg command failed: rc = -2 [ 5816.762275] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5816.762275] [ 5816.783631] LustreError: mgc192.168.204.137@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5816.826684] LustreError: 180935:0:(tgt_mount.c:2005:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5816.850743] LustreError: 180935:0:(tgt_mount.c:2799:server_fill_super()) Unable to start targets: -2 [ 5816.873562] LustreError: 180935:0:(tgt_mount.c:2249:server_put_super()) no obd lustre-MDT0000 [ 5816.886696] LustreError: 180935:0:(tgt_mount.c:130:server_deregister_mount()) lustre-MDT0000 not registered [ 5817.055088] Lustre: server umount lustre-MDT0000 complete [ 5817.071477] LustreError: 180935:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5828.502870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5828.864623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5833.556997] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5838.306486] LustreError: 181346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5838.331994] LustreError: 181346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5843.426158] LustreError: 181347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.798371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5844.324883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5849.166860] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 5857.017481] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5863.930226] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5870.048133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5870.060914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5870.065053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5870.074836] Lustre: Skipped 1 previous similar message [ 5870.095046] Lustre: Skipped 1 previous similar message [ 5875.171055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5875.191392] Lustre: Skipped 1 previous similar message [ 5880.287148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5880.292367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5880.295951] Lustre: Skipped 1 previous similar message [ 5880.479464] Lustre: server umount lustre-MDT0000 complete [ 5884.349318] LustreError: 181328:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788428403 with bad export cookie 3808631940209777023 [ 5884.377487] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5884.760118] Lustre: server umount lustre-MDT0001 complete [ 5894.326585] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5897.350912] Key type lgssc unregistered [ 5897.620620] LNet: 183670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5897.629185] LNetError: 183670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5897.652863] LNet: Removed LNI 192.168.204.137@tcp [ 5898.533176] Key type .llcrypt unregistered [ 5898.536754] Key type ._llcrypt unregistered [ 5919.931542] Key type ._llcrypt registered [ 5919.935964] Key type .llcrypt registered [ 5920.051245] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 5920.375415] Key type .llcrypt unregistered [ 5920.378484] Key type ._llcrypt unregistered [ 5938.518254] Key type ._llcrypt registered [ 5938.521694] Key type .llcrypt registered [ 5938.674328] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 5940.054277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5940.201478] alg: No test for adler32 (adler32-zlib) [ 5941.317346] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 5941.615282] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 5943.272483] Key type lgssc registered [ 5944.195486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5952.614935] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5952.627788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5953.734439] Lustre: 185807:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5969.247489] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5974.368497] LustreError: 185795:0:(mgc_request.c:317:do_config_log_add()) mgc192.168.204.137@tcp: failed processing log, type 4: rc = -110 [ 5979.104221] Lustre: 185810:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5984.671347] Lustre: 185795:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 5994.932561] Lustre: 185807:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 5997.663505] Lustre: 185812:0:(mgc_request.c:2378:mgc_process_log()) mgc192.168.204.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -110 [ 6000.098213] Lustre: 185810:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6004.192953] Lustre: 185807:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6005.151795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6007.967585] LustreError: 185812:0:(mgc_request.c:718:do_requeue()) failed processing log: -110 [ 6010.883643] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6014.436589] Lustre: 185807:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6014.456340] Lustre: 185807:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 1 previous similar message [ 6018.725996] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6026.605421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6026.877026] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6027.233412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6027.318048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6032.371277] LustreError: 185825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6032.413445] LustreError: 185825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6033.279638] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6037.474052] LustreError: 185826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6039.529743] LustreError: 186841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6039.560733] LustreError: 186841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6043.882077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6044.236952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6050.038155] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6060.183873] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6067.620165] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6070.723067] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6083.554281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6083.564322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6083.596333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6085.092266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6085.093227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6085.114249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6088.166400] Lustre: server umount lustre-OST0000 complete [ 6095.331333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6095.333749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6095.334015] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6095.343926] Lustre: Skipped 1 previous similar message [ 6095.345950] Lustre: Skipped 1 previous similar message [ 6098.356885] Lustre: server umount lustre-MDT0000 complete [ 6100.452612] LustreError: 186847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6100.466417] LustreError: 186847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6101.577501] LustreError: 188466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788428620 with bad export cookie 7259313810338855546 [ 6101.580160] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6101.595970] LustreError: 188466:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6102.008311] Lustre: server umount lustre-MDT0001 complete [ 6112.260556] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 6114.893812] Key type lgssc unregistered [ 6115.263720] LNet: 189770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6115.272084] LNetError: 189770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6115.286853] LNet: Removed LNI 192.168.204.137@tcp [ 6116.148652] Key type .llcrypt unregistered [ 6116.150804] Key type ._llcrypt unregistered [ 6138.391262] Key type ._llcrypt registered [ 6138.397549] Key type .llcrypt registered [ 6138.518230] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 6139.813558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6139.840895] alg: No test for adler32 (adler32-zlib) [ 6140.824509] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 6141.119194] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 6142.903224] Key type lgssc registered [ 6144.227973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6152.262792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6152.294513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6153.778543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6157.831876] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6163.938798] LustreError: 191179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6163.962210] LustreError: 191179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6166.891097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6167.311474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6172.033504] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6179.931515] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6186.474197] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6194.882123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6195.399742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6198.464381] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6202.952658] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6210.916738] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6217.182579] Lustre: Failing over lustre-MDT0000 [ 6218.211362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6218.231698] LustreError: 191907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6218.240522] Lustre: Skipped 1 previous similar message [ 6218.257361] LustreError: 191907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6219.429281] Lustre: server umount lustre-MDT0000 complete [ 6223.335402] LustreError: 191180:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6223.353163] LustreError: 191180:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6228.458681] LustreError: 193156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6228.481904] LustreError: 193156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6229.498846] LustreError: 191160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788428748 with bad export cookie 6031786943725410719 [ 6229.499656] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6229.516088] LustreError: 191160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6229.542631] Lustre: 193926:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6234.385058] LustreError: 193156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6234.419759] LustreError: 193156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6234.597341] Lustre: 193928:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6234.620891] Lustre: 193928:0:(ldlm_lib.c:1251:target_handle_connect()) Skipped 1 previous similar message [ 6239.718726] Lustre: 193928:0:(ldlm_lib.c:1251:target_handle_connect()) MGS: refusing connection from 0@lo: this is an MGS follower, which only serves clients that support it: rc = -19 [ 6240.398838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6240.742245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.816479] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6243.637575] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6245.890519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6245.924695] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6246.018484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6246.146934] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6251.397680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6257.448485] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6269.408150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6269.419801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6269.427228] Lustre: Skipped 1 previous similar message [ 6269.430925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6269.924861] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6269.931433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6271.461452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6274.810677] Lustre: server umount lustre-OST0000 complete [ 6281.710081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6281.711923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6281.716568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6281.716577] Lustre: Skipped 1 previous similar message [ 6281.732071] Lustre: Skipped 2 previous similar messages [ 6284.625824] Lustre: server umount lustre-MDT0000 complete [ 6286.817671] LustreError: 191179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6286.831023] LustreError: 191179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6288.185297] LustreError: 194763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788428807 with bad export cookie 6031786943725411783 [ 6288.193734] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6288.197117] LustreError: 194763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6288.620701] Lustre: server umount lustre-MDT0001 complete [ 6299.132735] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 6301.733280] Key type lgssc unregistered [ 6301.991699] LNet: 196022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6302.000189] LNetError: 196022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6302.024859] LNet: Removed LNI 192.168.204.137@tcp [ 6302.841691] Key type .llcrypt unregistered [ 6302.843821] Key type ._llcrypt unregistered [ 6324.918948] Key type ._llcrypt registered [ 6324.921771] Key type .llcrypt registered [ 6325.014658] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 6326.114080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6326.208302] alg: No test for adler32 (adler32-zlib) [ 6327.196970] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 6327.399731] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 6329.055201] Key type lgssc registered [ 6330.056375] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6338.223959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6338.243501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6339.739819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6343.970985] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6349.290812] LustreError: 197430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6349.311422] LustreError: 197430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6353.147353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6353.522378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6358.146242] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6365.509724] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6371.468732] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6378.673876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6378.991135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6385.158509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6385.828728] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6393.604965] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6397.783695] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6401.626829] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6421.070969] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6433.761202] Lustre: server umount lustre-OST0000 complete [ 6439.396321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6439.416884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6439.433452] Lustre: Skipped 1 previous similar message [ 6439.461772] Lustre: Skipped 1 previous similar message [ 6444.513062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6444.518421] Lustre: Skipped 1 previous similar message [ 6447.685817] Lustre: server umount lustre-MDT0000 complete [ 6449.636884] LustreError: 197429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6449.656791] LustreError: 197429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6450.591866] Lustre: 196742:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788428953/real 1788428953] req@00000000fb6b5ebb x1875303565676672/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788428969 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6450.635161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6450.654648] LustreError: 197430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6450.673181] LustreError: 197430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6451.720268] LustreError: 197409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788428970 with bad export cookie 17839158755937997576 [ 6451.723545] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6451.731635] LustreError: 197409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6452.298688] Lustre: server umount lustre-MDT0001 complete [ 6465.735071] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 6469.082400] Key type lgssc unregistered [ 6469.342152] LNet: 201298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6469.348890] LNetError: 201298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6470.383664] LNet: Removed LNI 192.168.204.137@tcp [ 6471.545348] Key type .llcrypt unregistered [ 6471.549048] Key type ._llcrypt unregistered [ 6499.362261] Key type ._llcrypt registered [ 6499.367430] Key type .llcrypt registered [ 6499.662755] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 6501.540761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6501.886117] alg: No test for adler32 (adler32-zlib) [ 6504.158994] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 6504.881558] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 6506.727295] Key type lgssc registered [ 6508.572490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6518.715038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6518.764665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6520.373303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6526.804865] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6530.546040] LustreError: 202561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6530.576795] LustreError: 202561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6535.657129] LustreError: 202562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6537.433871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6537.811125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6543.590974] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6552.990878] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6559.187804] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6566.653676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6567.214707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6574.674051] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6582.834930] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6590.773433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6599.205302] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6610.922900] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6616.153460] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6624.578702] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6637.032259] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6637.038732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6637.040979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6637.047603] Lustre: Skipped 1 previous similar message [ 6640.688080] Lustre: server umount lustre-OST0000 complete [ 6645.217322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6645.223224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6645.233335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6645.255044] Lustre: Skipped 1 previous similar message [ 6647.264464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6647.265621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6647.287584] Lustre: Skipped 1 previous similar message [ 6651.214363] Lustre: server umount lustre-MDT0000 complete [ 6652.393058] LustreError: 202556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6652.416806] LustreError: 202556:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6654.996475] LustreError: 204410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788429173 with bad export cookie 597175700699817153 [ 6655.000254] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6655.017946] LustreError: 204410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6655.450694] Lustre: server umount lustre-MDT0001 complete [ 6666.865725] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 6669.609337] Key type lgssc unregistered [ 6670.004279] LNet: 207412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6670.018094] LNetError: 207412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6670.036769] LNet: Removed LNI 192.168.204.137@tcp [ 6671.277335] Key type .llcrypt unregistered [ 6671.279434] Key type ._llcrypt unregistered [ 6696.809730] Key type ._llcrypt registered [ 6696.814523] Key type .llcrypt registered [ 6696.923433] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 6698.379898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6698.417480] alg: No test for adler32 (adler32-zlib) [ 6699.457634] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 6699.724571] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 6701.432686] Key type lgssc registered [ 6702.900059] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6711.159185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6711.183416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6712.842248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6718.444451] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6723.046026] LustreError: 208823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6723.071482] LustreError: 208823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6728.161225] LustreError: 208824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6728.872773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6729.248906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6734.969677] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6744.632310] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6751.756319] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6759.242583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6759.688858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6766.923523] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6776.129281] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6782.268366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6800.881344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6800.896348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6800.922162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6802.915570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6802.917389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6802.926653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6806.780757] Lustre: server umount lustre-OST0000 complete [ 6813.165486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6813.170465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6813.185975] Lustre: Skipped 1 previous similar message [ 6813.207333] Lustre: Skipped 2 previous similar messages [ 6818.287737] LustreError: 209552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6818.338041] LustreError: 209552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6818.460134] Lustre: server umount lustre-MDT0000 complete [ 6823.096945] LustreError: 208803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788429341 with bad export cookie 7186514697873791961 [ 6823.105193] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6823.117581] LustreError: 208803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6823.613211] Lustre: server umount lustre-MDT0001 complete [ 6836.488964] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 6840.374634] Key type lgssc unregistered [ 6840.789167] LNet: 212532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6840.804421] LNetError: 212532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6840.842199] LNet: Removed LNI 192.168.204.137@tcp [ 6841.933891] Key type .llcrypt unregistered [ 6841.935727] Key type ._llcrypt unregistered [ 6861.874426] Key type ._llcrypt registered [ 6861.877438] Key type .llcrypt registered [ 6861.988528] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 6863.223844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6863.372277] alg: No test for adler32 (adler32-zlib) [ 6864.445515] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 6864.625362] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 6866.263212] Key type lgssc registered [ 6867.458757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6876.364601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6876.392714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6877.943538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6882.715807] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6887.393249] LustreError: 213806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6887.433698] LustreError: 213806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6892.522719] LustreError: 213805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6892.693843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6893.211381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6898.827693] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6907.984563] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6914.180277] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6921.344820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6922.045327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6923.064698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6929.748795] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 6937.552683] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6952.932609] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6952.937640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6952.958532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6952.973850] Lustre: Skipped 1 previous similar message [ 6958.048813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6958.056463] Lustre: Skipped 2 previous similar messages [ 6958.842356] Lustre: server umount lustre-OST0000 complete [ 6964.705328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6964.711839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6964.731971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6968.303208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6968.306313] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6969.862290] Lustre: server umount lustre-MDT0000 complete [ 6973.412959] LustreError: 214534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6973.449032] LustreError: 214534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6973.956137] LustreError: 213786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788429492 with bad export cookie 8259415237437958410 [ 6973.964954] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6973.969710] LustreError: 213786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6974.366161] Lustre: server umount lustre-MDT0001 complete [ 6986.365734] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 6989.014186] Key type lgssc unregistered [ 6989.347790] LNet: 217492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6989.357351] LNetError: 217492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6990.439423] LNet: Removed LNI 192.168.204.137@tcp [ 6991.578343] Key type .llcrypt unregistered [ 6991.580489] Key type ._llcrypt unregistered [ 7013.100621] Key type ._llcrypt registered [ 7013.115659] Key type .llcrypt registered [ 7013.296673] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 7015.635092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7015.746357] alg: No test for adler32 (adler32-zlib) [ 7017.009581] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 7017.521566] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 7019.368696] Key type lgssc registered [ 7021.057596] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7031.264408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7031.298553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7032.886067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7037.796476] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7043.047363] LustreError: 218755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.057642] LustreError: 218755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7046.919460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7047.249196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7051.649905] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7058.964639] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7064.741356] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7071.032878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7071.496858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7077.680422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7077.806565] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7084.720847] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7094.319665] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7094.407631] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7094.539627] LustreError: 220576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7094.681713] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7100.289918] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7107.703521] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7109.238429] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7120.977560] Lustre: Permanently deactivating lustre-OST0001 [ 7120.999708] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7128.424635] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7135.261117] Lustre: Permanently reactivating lustre-OST0001 [ 7135.273396] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7135.277970] Lustre: Skipped 2 previous similar messages [ 7142.171512] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7142.187412] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7142.200207] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7142.222485] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7142.297892] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7149.045660] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7149.061171] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7149.066588] Lustre: Skipped 2 previous similar messages [ 7149.083977] Lustre: Skipped 1 previous similar message [ 7154.164126] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7154.172118] Lustre: Skipped 1 previous similar message [ 7159.275385] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7159.279935] Lustre: Skipped 1 previous similar message [ 7163.359294] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7163.483502] Lustre: server umount lustre-OST0001 complete [ 7164.397245] LustreError: 220577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7164.410603] LustreError: 220577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7169.512265] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7169.512474] LustreError: 221888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7169.518070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7169.525965] Lustre: Skipped 1 previous similar message [ 7169.545254] LustreError: 221888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7173.572563] Lustre: server umount lustre-OST0000 complete [ 7179.746207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7179.752767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7179.760484] Lustre: Skipped 1 previous similar message [ 7179.774415] Lustre: Skipped 2 previous similar messages [ 7183.602563] Lustre: server umount lustre-MDT0000 complete [ 7184.864716] LustreError: 218755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7184.890185] LustreError: 218755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7187.913719] LustreError: 218736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788429706 with bad export cookie 4463680812364655147 [ 7187.916450] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7187.924813] LustreError: 218736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7188.297144] Lustre: server umount lustre-MDT0001 complete [ 7198.448736] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 7201.386815] Key type lgssc unregistered [ 7201.740544] LNet: 223836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7201.752189] LNetError: 223836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7201.781779] LNet: Removed LNI 192.168.204.137@tcp [ 7202.908313] Key type .llcrypt unregistered [ 7202.911762] Key type ._llcrypt unregistered [ 7243.713809] Key type ._llcrypt registered [ 7243.719322] Key type .llcrypt registered [ 7243.825891] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 7244.885966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7244.944800] alg: No test for adler32 (adler32-zlib) [ 7246.019623] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 7246.256436] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 7247.936251] Key type lgssc registered [ 7249.064763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7257.433666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7257.455856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7258.821567] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7258.863772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7259.104700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7264.602491] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7273.553046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7273.642037] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7273.677561] Lustre: 226040:0:(mgs_llog.c:1456:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7273.971806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7278.529262] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7285.815854] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7291.938599] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7297.847280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7298.022882] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7298.274174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7303.771275] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7304.940844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7311.229943] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7315.306631] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7322.160717] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7329.803706] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7338.591599] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7347.572158] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7365.359982] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7365.371263] Lustre: Skipped 1 previous similar message [ 7400.001909] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7400.007322] Lustre: Skipped 3 previous similar messages [ 7468.593793] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7468.605894] Lustre: Skipped 7 previous similar messages [ 7510.000649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7510.003786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7510.023405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7510.032829] Lustre: Skipped 1 previous similar message [ 7510.063034] Lustre: Skipped 1 previous similar message [ 7515.107140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7515.116081] Lustre: Skipped 1 previous similar message [ 7515.536946] Lustre: server umount lustre-OST0000 complete [ 7524.835749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7524.850182] LustreError: Skipped 1 previous similar message [ 7524.860325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7524.878644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7527.100628] Lustre: server umount lustre-MDT0000 complete [ 7531.661147] LustreError: 228605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788430050 with bad export cookie 3997950870380074911 [ 7531.675645] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7531.686843] LustreError: 228605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7532.175517] Lustre: server umount lustre-MDT0001 complete [ 7542.708310] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 7545.887951] Key type lgssc unregistered [ 7546.157887] LNet: 230866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7546.172301] LNetError: 230866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7546.191742] LNet: Removed LNI 192.168.204.137@tcp [ 7547.149319] Key type .llcrypt unregistered [ 7547.154626] Key type ._llcrypt unregistered [ 7570.490558] Key type ._llcrypt registered [ 7570.493247] Key type .llcrypt registered [ 7570.591605] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 7572.170619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7572.209251] alg: No test for adler32 (adler32-zlib) [ 7573.367751] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 7573.785864] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 7575.495171] Key type lgssc registered [ 7576.886849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7584.371966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7584.398155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7585.788488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7590.595670] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7596.006937] LustreError: 232278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7596.028256] LustreError: 232278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7599.498467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7599.969361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7604.920508] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7613.709892] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7621.626279] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7628.320526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7628.579695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7630.658481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7635.017074] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_default_debug -1 all [ 7643.377879] Lustre: DEBUG MARKER: oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7665.650373] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7665.656993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7665.659087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7665.697403] Lustre: Skipped 1 previous similar message [ 7668.830761] Lustre: server umount lustre-OST0000 complete [ 7675.873750] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7675.879537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7675.893454] Lustre: Skipped 1 previous similar message [ 7675.899569] Lustre: Skipped 1 previous similar message [ 7678.996632] Lustre: server umount lustre-MDT0000 complete [ 7680.995912] LustreError: 234222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7681.020855] LustreError: 234222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7682.781366] LustreError: 232258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788430201 with bad export cookie 14948836807656182313 [ 7682.786436] LustreError: mgc192.168.204.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7682.799023] LustreError: 232258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7683.153980] Lustre: server umount lustre-MDT0001 complete [ 7694.109173] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 7696.847872] Key type lgssc unregistered [ 7697.229231] LNet: 235968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7697.244133] LNetError: 235968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7697.260092] LNet: Removed LNI 192.168.204.137@tcp [ 7698.380133] Key type .llcrypt unregistered [ 7698.383291] Key type ._llcrypt unregistered [ 7720.378395] Key type ._llcrypt registered [ 7720.380450] Key type .llcrypt registered [ 7720.486139] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 7721.526124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7721.706880] alg: No test for adler32 (adler32-zlib) [ 7722.772738] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 7723.000213] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 7724.703882] Key type lgssc registered [ 7726.393538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7997.358890] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 8000.167882] Key type lgssc unregistered [ 8000.512640] LNet: 237988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8000.516899] LNetError: 237988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8000.546489] LNet: Removed LNI 192.168.204.137@tcp [ 8001.399800] Key type .llcrypt unregistered [ 8001.402085] Key type ._llcrypt unregistered [ 8026.674988] Key type ._llcrypt registered [ 8026.677381] Key type .llcrypt registered [ 8026.833470] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing set_hostid [ 8041.216292] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing load_modules_local [ 8042.405916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8042.447920] alg: No test for adler32 (adler32-zlib) [ 8043.563545] Lustre: Lustre: Build Version: 2.17.58_7_g065ac5e [ 8043.933528] LNet: Added LNI 192.168.204.137@tcp [8/256/0/180] [ 8045.656393] Key type lgssc registered [ 8046.751242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8078.467947] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7800 sec ========== 06:16:34 (1788430594) [ 8080.231551] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:16:36 (1788430596) === [ 8084.781532] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:16:40 (1788430600) === [ 8128.540828] Lustre: DEBUG MARKER: oleg437-server.virtnet: executing unload_modules_local [ 8132.050448] Key type lgssc unregistered [ 8132.494822] LNet: 243219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8132.521736] LNetError: 243219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8132.539994] LNet: Removed LNI 192.168.204.137@tcp [ 8133.898579] Key type .llcrypt unregistered [ 8133.902763] Key type ._llcrypt unregistered