[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 470299948 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002275] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006819] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.010085] LSM: Security Framework initializing [ 0.011056] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014073] *** VALIDATE selinux *** [ 0.022147] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027098] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030074] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032034] *** VALIDATE tmpfs *** [ 0.033412] *** VALIDATE proc *** [ 0.035086] *** VALIDATE cgroup *** [ 0.036000] *** VALIDATE cgroup2 *** [ 0.036267] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041028] Spectre V2 : User space: Vulnerable [ 0.042010] Speculative Store Bypass: Vulnerable [ 0.044879] debug: unmapping init [mem 0xffffffffa6859000-0xffffffffa6860fff] [ 0.046180] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047737] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048027] ... version: 2 [ 0.049014] ... bit width: 48 [ 0.050013] ... generic registers: 4 [ 0.051017] ... value mask: 0000ffffffffffff [ 0.052020] ... max period: 00007fffffffffff [ 0.053017] ... fixed-purpose events: 3 [ 0.054015] ... event mask: 000000070000000f [ 0.055325] rcu: Hierarchical SRCU implementation. [ 0.057578] smp: Bringing up secondary CPUs ... [ 0.058583] x86: Booting SMP configuration: [ 0.059065] .... node #0, CPUs: #1 #2 #3 [ 0.062013] smp: Brought up 1 node, 4 CPUs [ 0.064016] smpboot: Max logical packages: 1 [ 0.065010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.214797] node 0 deferred pages initialised in 148ms [ 0.218288] devtmpfs: initialized [ 0.219305] x86/mm: Memory block size: 128MB [ 0.221877] gcov: version magic: 0x41383552 [ 0.224138] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.225115] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.226341] pinctrl core: initialized pinctrl subsystem [ 0.227326] [ 0.227978] ************************************************************* [ 0.228021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.229022] ** ** [ 0.230018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.231024] ** ** [ 0.232013] ** This means that this kernel is built to expose internal ** [ 0.233021] ** IOMMU data structures, which may compromise security on ** [ 0.234020] ** your system. ** [ 0.235012] ** ** [ 0.236016] ** If you see this message and you are not debugging the ** [ 0.237022] ** kernel, report this immediately to your vendor! ** [ 0.238031] ** ** [ 0.239018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.240019] ************************************************************* [ 0.241712] NET: Registered protocol family 16 [ 0.242480] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.243094] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.244070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.245626] cpuidle: using governor menu [ 0.247939] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.252515] PCI: Using configuration type 1 for base access [ 0.255134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.267072] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.269021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.274036] cryptd: max_cpu_qlen set to 1000 [ 0.277275] ACPI: Added _OSI(Module Device) [ 0.279015] ACPI: Added _OSI(Processor Device) [ 0.280014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.282019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.287850] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.294031] ACPI: Interpreter enabled [ 0.296076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.298019] ACPI: Using IOAPIC for interrupt routing [ 0.300111] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.305384] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.318850] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.322047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.325024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.329175] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.334441] acpiphp: Slot [2] registered [ 0.336124] acpiphp: Slot [5] registered [ 0.338183] acpiphp: Slot [6] registered [ 0.339121] acpiphp: Slot [7] registered [ 0.340083] acpiphp: Slot [8] registered [ 0.341082] acpiphp: Slot [9] registered [ 0.342102] acpiphp: Slot [10] registered [ 0.344096] acpiphp: Slot [3] registered [ 0.345072] acpiphp: Slot [4] registered [ 0.346106] acpiphp: Slot [11] registered [ 0.347113] acpiphp: Slot [12] registered [ 0.349106] acpiphp: Slot [13] registered [ 0.350109] acpiphp: Slot [14] registered [ 0.352123] acpiphp: Slot [15] registered [ 0.353122] acpiphp: Slot [16] registered [ 0.355119] acpiphp: Slot [17] registered [ 0.357122] acpiphp: Slot [18] registered [ 0.358137] acpiphp: Slot [19] registered [ 0.359095] acpiphp: Slot [20] registered [ 0.361109] acpiphp: Slot [21] registered [ 0.362116] acpiphp: Slot [22] registered [ 0.364091] acpiphp: Slot [23] registered [ 0.365111] acpiphp: Slot [24] registered [ 0.367113] acpiphp: Slot [25] registered [ 0.368117] acpiphp: Slot [26] registered [ 0.370125] acpiphp: Slot [27] registered [ 0.372110] acpiphp: Slot [28] registered [ 0.373146] acpiphp: Slot [29] registered [ 0.374121] acpiphp: Slot [30] registered [ 0.376089] acpiphp: Slot [31] registered [ 0.377059] PCI host bridge to bus 0000:00 [ 0.379020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.381032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.384025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.387041] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.390035] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.393034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.395203] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.398011] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.401349] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.413017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.417020] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.420019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.422027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.427026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.430594] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.434185] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.436047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.439679] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.444021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.456782] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.461920] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.466335] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.479019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.495026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.535020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.559816] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.566018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.576020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.594020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.604925] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.615016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.621016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.635019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.641016] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.645017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.649018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.662020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.671412] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.678021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.684017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.701017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.708536] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.712016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.717024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.726014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.733681] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.735244] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.737296] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.738277] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.740203] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.744116] iommu: Default domain type: Passthrough [ 0.746360] SCSI subsystem initialized [ 0.747091] ACPI: bus type USB registered [ 0.747947] usbcore: registered new interface driver usbfs [ 0.749072] usbcore: registered new interface driver hub [ 0.752099] usbcore: registered new device driver usb [ 0.754178] pps_core: LinuxPPS API ver. 1 registered [ 0.756014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.759071] PTP clock support registered [ 0.761047] EDAC MC: Ver: 3.0.0 [ 0.763007] PCI: Using ACPI for IRQ routing [ 0.764958] NetLabel: Initializing [ 0.766014] NetLabel: domain hash size = 128 [ 0.768012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.770127] NetLabel: unlabeled traffic allowed by default [ 0.773092] vgaarb: loaded [ 0.774260] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.776017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.784721] clocksource: Switched to clocksource kvm-clock [ 0.887867] VFS: Disk quotas dquot_6.6.0 [ 0.889609] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.892338] *** VALIDATE ramfs *** [ 0.894013] *** VALIDATE hugetlbfs *** [ 0.896136] pnp: PnP ACPI init [ 0.898859] pnp: PnP ACPI: found 6 devices [ 0.918626] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.922338] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.924337] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.926273] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.928271] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.930279] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.932452] NET: Registered protocol family 2 [ 0.934773] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.940454] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.943616] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.948462] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.952703] TCP: Hash tables configured (established 65536 bind 65536) [ 0.955869] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.958558] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.960685] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.963090] NET: Registered protocol family 1 [ 0.965104] RPC: Registered named UNIX socket transport module. [ 0.967480] RPC: Registered udp transport module. [ 0.969219] RPC: Registered tcp transport module. [ 0.971170] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.973964] NET: Registered protocol family 44 [ 0.975789] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.978311] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.980502] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.982726] PCI: CLS 0 bytes, default 64 [ 0.984349] Unpacking initramfs... [ 2.273392] debug: unmapping init [mem 0xffff9aa8bcc54000-0xffff9aa8bffbffff] [ 2.278536] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.281819] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.285823] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.739930] Initialise system trusted keyrings [ 2.742367] Key type blacklist registered [ 2.745132] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.753176] zbud: loaded [ 2.756297] *** VALIDATE nfs *** [ 2.757445] *** VALIDATE nfs4 *** [ 2.759284] pstore: using deflate compression [ 2.762355] Platform Keyring initialized [ 2.843380] NET: Registered protocol family 38 [ 2.845324] Key type asymmetric registered [ 2.849233] Asymmetric key parser 'x509' registered [ 2.850987] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.853810] io scheduler mq-deadline registered [ 2.855564] io scheduler kyber registered [ 2.857574] io scheduler bfq registered [ 2.859618] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.862947] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.866143] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.869527] ACPI: Power Button [PWRF] [ 2.874645] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.879726] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.891256] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.897367] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.911387] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.940863] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.967980] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.973993] Non-volatile memory driver v1.3 [ 2.975616] Linux agpgart interface v0.103 [ 3.006651] virtio_blk virtio1: [vda] 149776 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.009387] vda: detected capacity change from 0 to 76685312 [ 3.023524] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.026467] vdb: detected capacity change from 0 to 1073741824 [ 3.043873] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.048477] vdc: detected capacity change from 0 to 2621440000 [ 3.071973] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.074861] vdd: detected capacity change from 0 to 2621440000 [ 3.092236] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.095012] vde: detected capacity change from 0 to 4294967296 [ 3.110788] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.114297] vdf: detected capacity change from 0 to 4294967296 [ 3.121957] libphy: Fixed MDIO Bus: probed [ 3.127990] usbcore: registered new interface driver usbserial_generic [ 3.131080] usbserial: USB Serial support registered for generic [ 3.133502] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.138144] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.140225] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.142743] mousedev: PS/2 mouse device common for all mice [ 3.145798] rtc_cmos 00:05: RTC can wake from S4 [ 3.148311] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.152183] rtc_cmos 00:05: registered as rtc0 [ 3.154377] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.154978] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.158564] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.160273] intel_pstate: CPU model not supported [ 3.166245] hid: raw HID events driver (C) Jiri Kosina [ 3.168244] usbcore: registered new interface driver usbhid [ 3.169961] usbhid: USB HID core driver [ 3.171253] drop_monitor: Initializing network drop monitor service [ 3.173487] Initializing XFRM netlink socket [ 3.175213] NET: Registered protocol family 10 [ 3.178095] Segment Routing with IPv6 [ 3.179738] NET: Registered protocol family 17 [ 3.181492] mpls_gso: MPLS GSO support [ 3.186532] RAS: Correctable Errors collector initialized. [ 3.188789] AVX version of gcm_enc/dec engaged. [ 3.190391] AES CTR mode by8 optimization enabled [ 3.263047] sched_clock: Marking stable (3263018735, 0)->(4178656694, -915637959) [ 3.266583] registered taskstats version 1 [ 3.268373] Loading compiled-in X.509 certificates [ 3.270279] zswap: loaded using pool lzo/zbud [ 3.296833] Key type big_key registered [ 3.310641] Key type encrypted registered [ 3.312996] ima: No TPM chip found, activating TPM-bypass! [ 3.315556] ima: Allocated hash algorithm: sha1 [ 3.317747] ima: No architecture policies found [ 3.319488] evm: Initialising EVM extended attributes: [ 3.321386] evm: security.selinux [ 3.322636] evm: security.ima [ 3.323809] evm: security.capability [ 3.325150] evm: HMAC attrs: 0x1 [ 3.327963] rtc_cmos 00:05: setting system clock to 2026-09-05 05:23:06 UTC (1788585786) [ 3.333826] debug: unmapping init [mem 0xffffffffa7803000-0xffffffffa79fffff] [ 3.336803] debug: unmapping init [mem 0xffffffffa6582000-0xffffffffa6858fff] [ 3.345102] Write protecting the kernel read-only data: 28672k [ 3.348827] debug: unmapping init [mem 0xffffffffa4c03000-0xffffffffa4dfffff] [ 3.351644] debug: unmapping init [mem 0xffffffffa5514000-0xffffffffa55fffff] [ 3.385517] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.394617] systemd[1]: Detected virtualization kvm. [ 3.396968] systemd[1]: Detected architecture x86-64. [ 3.399408] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.427830] systemd[1]: No hostname configured. [ 3.429931] systemd[1]: Set hostname to . [ 3.432324] random: systemd: uninitialized urandom read (16 bytes read) [ 3.435415] systemd[1]: Initializing machine ID from random generator. [ 3.591286] random: systemd: uninitialized urandom read (16 bytes read) [ 3.606063] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.618169] random: systemd: uninitialized urandom read (16 bytes read) [ 3.621318] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.629643] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.286734] device-mapper: uevent: version 1.0.3 [ 4.288608] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.987866] virtio_net virtio0 ens2: renamed from eth0 [ 5.008385] scsi host0: ata_piix [ 5.011595] random: fast init done [ 5.027823] scsi host1: ata_piix [ 5.030444] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.035202] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.467417] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.830542] random: crng init done [ 9.831748] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.305614] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.477484] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.721965] SELinux: Disabled at runtime. [ 11.786340] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.795442] systemd[1]: Detected virtualization kvm. [ 11.797465] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.301358] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.304151] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.308967] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.314547] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.317491] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.325670] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.330137] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. [ 12.523995] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.848782] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.145537] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.151627] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.324576] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.336638] EDAC sbridge: Ver: 1.1.2 [ 15.128044] Key type dns_resolver registered [ 15.428402] NFS: Registering the id_resolver key type [ 15.430796] Key type id_resolver registered [ 15.432890] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg209-server login: [ 40.996323] libcfs: loading out-of-tree module taints kernel. [ 41.016622] Key type ._llcrypt registered [ 41.018385] Key type .llcrypt registered [ 41.065796] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 52.706896] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 54.463577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 54.492902] alg: No test for adler32 (adler32-zlib) [ 56.032256] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 57.225356] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 59.089369] Key type lgssc registered [ 61.351731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 86.501620] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 112.048920] hrtimer: interrupt took 8909659 ns [ 138.156095] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 155.310335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 155.395729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 156.595709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 156.626549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 156.722576] Lustre: lustre-MDT0000: new disk, initializing [ 156.809963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 156.834699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 162.099865] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.377773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.517240] Lustre: 6485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 178.549243] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 178.553811] Lustre: Skipped 1 previous similar message [ 178.646347] Lustre: lustre-MDT0001: new disk, initializing [ 178.717959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 178.753623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 178.771577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.654947] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.289996] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 199.980808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 200.370290] Lustre: lustre-OST0000: new disk, initializing [ 200.374974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 200.386872] Lustre: 8422:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 200.456370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.219267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.246958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 204.370638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 206.453449] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.197595] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 222.398496] Lustre: lustre-OST0001: new disk, initializing [ 222.403900] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 222.413639] Lustre: 9494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.501058] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 227.897892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 227.906618] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 228.014942] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 230.046310] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 242.680564] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 250.470133] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 258.104775] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing check_logdir /tmp/testlogs/ [ 264.118906] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing yml_node [ 269.346573] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 272.256250] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 275.251137] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 276.639240] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 01:27:39 EDT 2026 [ 298.876670] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 300.658280] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 319.969073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.972169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 319.985229] Lustre: Skipped 1 previous similar message [ 319.997556] Lustre: Skipped 3 previous similar messages [ 324.602610] Lustre: server umount lustre-MDT0000 complete [ 328.935736] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586112 with bad export cookie 11864183014248506495 [ 328.942858] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 328.963977] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 330.210153] LustreError: 6491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 330.210892] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.220138] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 330.252040] LustreError: 6491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 330.253628] Lustre: Skipped 3 previous similar messages [ 333.217131] LustreError: 6490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 333.217974] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 333.233458] Lustre: Skipped 2 previous similar messages [ 335.488359] Lustre: server umount lustre-MDT0001 complete [ 339.146369] Lustre: server umount lustre-OST0000 complete [ 343.064976] Lustre: server umount lustre-OST0001 complete [ 350.124837] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 360.069128] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 395.976596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 396.175298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 396.203522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 396.254234] Lustre: lustre-MDT0000: new disk, initializing [ 396.306637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 396.316581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 400.279667] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 411.843707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.974388] Lustre: 15050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 412.031301] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 412.036423] Lustre: Skipped 1 previous similar message [ 412.100911] Lustre: lustre-MDT0001: new disk, initializing [ 412.203949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 412.237053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 412.253858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 417.154897] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 427.138807] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 433.436240] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 441.221192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 441.680580] Lustre: lustre-OST0000: new disk, initializing [ 441.689973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 441.702848] Lustre: 17015:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 441.856606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.210290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 443.224962] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 443.395389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 447.547231] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 457.988342] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 462.497848] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 462.730376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 468.771040] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 469.072736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 474.082615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 474.092809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.118443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 474.121049] Lustre: Skipped 1 previous similar message [ 479.206215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 479.228185] Lustre: Skipped 1 previous similar message [ 485.344145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 485.492472] Lustre: server umount lustre-OST0000 complete [ 489.441594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.450395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 489.459844] Lustre: Skipped 2 previous similar messages [ 489.476825] Lustre: Skipped 3 previous similar messages [ 496.072698] Lustre: server umount lustre-MDT0000 complete [ 499.684806] LustreError: 15057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.698173] LustreError: 15057:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 499.938370] LustreError: 15041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586283 with bad export cookie 11864183014248508028 [ 499.948724] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 499.966892] LustreError: 15041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 500.319749] Lustre: server umount lustre-MDT0001 complete [ 503.898245] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 01:31:25 (1788586285) [ 511.717488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.168182] LustreError: 19284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 512.227590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 516.385949] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 517.603354] LustreError: 19285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.879493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 533.905257] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 542.947206] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 549.019097] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 557.145041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 564.545444] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 572.838604] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 593.379968] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 593.393937] LustreError: Skipped 1 previous similar message [ 593.401898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.421965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 593.433051] Lustre: Skipped 2 previous similar messages [ 597.476557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 598.866522] Lustre: server umount lustre-OST0000 complete [ 602.599759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 602.620614] Lustre: Skipped 2 previous similar messages [ 608.773662] Lustre: server umount lustre-MDT0000 complete [ 612.835175] LustreError: 20208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 612.853978] LustreError: 20208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 613.527037] LustreError: 21127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586396 with bad export cookie 11864183014248508910 [ 613.531227] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 613.544616] LustreError: 21127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 614.033650] Lustre: server umount lustre-MDT0001 complete [ 628.002804] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 631.192964] Key type lgssc unregistered [ 631.524733] LNet: 22839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 631.536627] LNetError: 22839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 631.554341] LNet: Removed LNI 192.168.202.109@tcp [ 632.544175] Key type .llcrypt unregistered [ 632.546909] Key type ._llcrypt unregistered [ 658.949762] Key type ._llcrypt registered [ 658.953240] Key type .llcrypt registered [ 659.051581] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 660.824888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 660.874344] alg: No test for adler32 (adler32-zlib) [ 661.916197] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 662.126183] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 663.889669] Key type lgssc registered [ 665.096320] Lustre: Echo OBD driver; http://www.lustre.org/ [ 673.609186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 673.654265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 675.336494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 681.346977] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 685.544147] LustreError: 24249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 685.575583] LustreError: 24249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 690.323259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 690.670623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 695.039594] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 703.016488] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 709.089189] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 716.416034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 716.973994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 720.051647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 723.821374] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 732.982512] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 751.088892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 751.092891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 751.104424] LustreError: Skipped 1 previous similar message [ 751.106541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 751.140536] Lustre: Skipped 1 previous similar message [ 756.196426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 756.202044] Lustre: Skipped 2 previous similar messages [ 761.316972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 761.328050] Lustre: Skipped 1 previous similar message [ 763.872582] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 764.110180] Lustre: server umount lustre-OST0000 complete [ 771.553534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 771.558097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 771.577380] Lustre: Skipped 1 previous similar message [ 771.596428] Lustre: Skipped 1 previous similar message [ 775.370140] Lustre: server umount lustre-MDT0000 complete [ 776.677519] LustreError: 24249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 776.717265] LustreError: 24249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 779.929413] LustreError: 24228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586563 with bad export cookie 17891416224826609127 [ 779.942368] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 780.290948] Lustre: server umount lustre-MDT0001 complete [ 793.469355] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 796.186884] Key type lgssc unregistered [ 796.524193] LNet: 28226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.535380] LNetError: 28226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 797.608084] LNet: Removed LNI 192.168.202.109@tcp [ 798.930895] Key type .llcrypt unregistered [ 798.937605] Key type ._llcrypt unregistered [ 828.181658] Key type ._llcrypt registered [ 828.183464] Key type .llcrypt registered [ 828.277184] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 829.641518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 829.727748] alg: No test for adler32 (adler32-zlib) [ 830.826641] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 831.144517] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 832.937277] Key type lgssc registered [ 834.152815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 842.659733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 842.697458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 844.372260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 849.512585] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 854.517445] LustreError: 29636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 854.560757] LustreError: 29636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 859.630168] LustreError: 29637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 859.771624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.379970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 865.079413] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 873.740505] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 881.669369] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 898.342659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 898.971848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 905.465243] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 913.567989] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 914.308914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 924.653590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 924.655037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 924.703939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 924.716499] Lustre: Skipped 1 previous similar message [ 929.764981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 929.775417] Lustre: Skipped 1 previous similar message [ 930.519955] Lustre: server umount lustre-OST0000 complete [ 934.884433] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.899186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 934.900112] Lustre: Skipped 2 previous similar messages [ 940.007318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 940.018668] Lustre: Skipped 2 previous similar messages [ 945.123644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 945.132052] Lustre: Skipped 1 previous similar message [ 948.193702] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 948.438421] Lustre: server umount lustre-MDT0000 complete [ 950.241255] LustreError: 29637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 950.279602] LustreError: 29637:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 952.935825] LustreError: 29618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586736 with bad export cookie 11211542580995858812 [ 952.953072] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 953.281792] Lustre: server umount lustre-MDT0001 complete [ 963.870744] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 966.010833] Key type lgssc unregistered [ 966.262663] LNet: 33519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.272412] LNetError: 33519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.335423] LNet: Removed LNI 192.168.202.109@tcp [ 968.233632] Key type .llcrypt unregistered [ 968.235782] Key type ._llcrypt unregistered [ 989.317278] Key type ._llcrypt registered [ 989.319350] Key type .llcrypt registered [ 989.412668] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 990.516843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 990.546902] alg: No test for adler32 (adler32-zlib) [ 991.547107] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 991.746459] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 993.392179] Key type lgssc registered [ 994.509716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1002.070254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1002.085394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1003.640334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1008.165721] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1013.219646] LustreError: 34907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1013.255397] LustreError: 34907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1017.442526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1017.987124] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1023.981185] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1032.588835] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1038.602073] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1045.303233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1045.750893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1048.856429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1052.479464] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1061.413470] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1074.660326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1074.669728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1074.674253] LustreError: Skipped 1 previous similar message [ 1074.695958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1078.755102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1078.767910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1078.769126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1080.768488] Lustre: server umount lustre-OST0000 complete [ 1088.997536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.023513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1089.024241] Lustre: Skipped 1 previous similar message [ 1089.051877] Lustre: Skipped 2 previous similar messages [ 1092.083749] Lustre: server umount lustre-MDT0000 complete [ 1094.117430] LustreError: 34903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1094.143131] LustreError: 34903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1096.415019] LustreError: 34890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788586879 with bad export cookie 14061594636319226236 [ 1096.429600] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1096.741792] Lustre: server umount lustre-MDT0001 complete [ 1108.419941] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1111.765863] Key type lgssc unregistered [ 1112.107640] LNet: 38466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.121006] LNetError: 38466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1112.142581] LNet: Removed LNI 192.168.202.109@tcp [ 1112.969813] Key type .llcrypt unregistered [ 1112.975424] Key type ._llcrypt unregistered [ 1135.528817] Key type ._llcrypt registered [ 1135.532890] Key type .llcrypt registered [ 1135.640558] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1136.972485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1137.036902] alg: No test for adler32 (adler32-zlib) [ 1138.178426] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 1138.430588] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1140.104456] Key type lgssc registered [ 1141.085786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1148.722199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1148.747709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1150.435478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1156.017830] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1160.683426] LustreError: 39876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.718649] LustreError: 39876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1165.806957] LustreError: 39877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1166.357423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1166.826698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1171.043925] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1178.615594] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1184.433284] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1191.241911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1191.340138] LustreError: 41696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1191.585615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1195.715608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1197.614642] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1205.572954] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1213.355560] Lustre: lustre-OST0000: Not available for connect from 192.168.202.9@tcp (stopping) [ 1216.485032] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1216.485244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1216.490052] LustreError: Skipped 1 previous similar message [ 1216.500156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1216.511222] Lustre: Skipped 1 previous similar message [ 1216.524451] Lustre: Skipped 1 previous similar message [ 1218.511899] Lustre: server umount lustre-OST0000 complete [ 1241.072056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.073550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.088464] Lustre: Skipped 1 previous similar message [ 1241.101306] Lustre: Skipped 1 previous similar message [ 1245.034681] Lustre: server umount lustre-MDT0000 complete [ 1246.192204] LustreError: 39877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1246.228031] LustreError: 39877:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1249.490340] LustreError: 39858:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587032 with bad export cookie 4904503083027926457 [ 1249.501708] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1255.943182] Lustre: server umount lustre-MDT0001 complete [ 1269.420862] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1272.295560] Key type lgssc unregistered [ 1272.573796] LNet: 43587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1272.590432] LNetError: 43587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1272.614813] LNet: Removed LNI 192.168.202.109@tcp [ 1273.396656] Key type .llcrypt unregistered [ 1273.399705] Key type ._llcrypt unregistered [ 1298.305807] Key type ._llcrypt registered [ 1298.310024] Key type .llcrypt registered [ 1298.431978] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1299.628396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1299.746752] alg: No test for adler32 (adler32-zlib) [ 1300.760474] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 1301.032828] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1302.713419] Key type lgssc registered [ 1303.832579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1312.711987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1312.732135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1314.173321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.672670] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1324.516816] LustreError: 44996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1324.529209] LustreError: 44996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1327.490720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1327.890874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1332.254795] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1340.016753] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1346.410824] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1354.854471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1355.336954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1359.426911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1362.961320] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1371.563093] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1379.257147] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.9@tcp (stopping) [ 1379.299413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1379.304219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.902536] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.912128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1382.924667] Lustre: Skipped 3 previous similar messages [ 1384.332383] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.9@tcp (stopping) [ 1384.643604] Lustre: server umount lustre-MDT0000 complete [ 1388.015617] LustreError: 44991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.029149] LustreError: 44991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1388.666274] LustreError: 44978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587171 with bad export cookie 8012749470403168056 [ 1388.676672] LustreError: 44978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1388.676848] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1389.496176] LustreError: 46045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1389.515814] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.9@tcp (stopping) [ 1393.122687] LustreError: 44991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1393.140063] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.170518] Lustre: Skipped 1 previous similar message [ 1394.573357] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.9@tcp (stopping) [ 1394.580073] Lustre: Skipped 1 previous similar message [ 1398.241636] LustreError: 44991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.279921] LustreError: 44991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1403.360178] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1403.753426] Lustre: server umount lustre-MDT0001 complete [ 1420.042314] Lustre: server umount lustre-OST0000 complete [ 1438.825559] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1441.676520] Key type lgssc unregistered [ 1442.020240] LNet: 48858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1442.026846] LNetError: 48858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1442.042850] LNet: Removed LNI 192.168.202.109@tcp [ 1442.881956] Key type .llcrypt unregistered [ 1442.895583] Key type ._llcrypt unregistered [ 1472.242196] Key type ._llcrypt registered [ 1472.245877] Key type .llcrypt registered [ 1472.395268] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1473.527486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1473.580276] alg: No test for adler32 (adler32-zlib) [ 1474.764172] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 1475.079600] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1476.811322] Key type lgssc registered [ 1478.129507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1486.113921] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1486.143848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1501.728473] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1506.848347] LustreError: 50243:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 1533.538478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1539.156407] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1547.013864] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1712.797636] Lustre: server umount lustre-OST0000 complete [ 1731.594831] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1734.884586] Key type lgssc unregistered [ 1735.225366] LNet: 51922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.245509] LNetError: 51922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1735.263294] LNet: Removed LNI 192.168.202.109@tcp [ 1736.031560] Key type .llcrypt unregistered [ 1736.032703] Key type ._llcrypt unregistered [ 1761.798447] Key type ._llcrypt registered [ 1761.801719] Key type .llcrypt registered [ 1761.902779] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1763.094193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1763.271375] alg: No test for adler32 (adler32-zlib) [ 1764.320546] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 1764.703377] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1766.392179] Key type lgssc registered [ 1767.281141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1775.294581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1775.308946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1776.731875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1780.790437] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1786.857591] LustreError: 53329:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1786.880573] LustreError: 53329:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1789.477903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1789.883228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1793.826539] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1801.451388] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1808.395681] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1815.712098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1816.040422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1822.206705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1823.018931] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1831.006234] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1834.745079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1834.748201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1840.109863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1840.121523] Lustre: Skipped 2 previous similar messages [ 1845.217322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1845.220505] Lustre: Skipped 1 previous similar message [ 1848.288268] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1848.464538] Lustre: server umount lustre-OST0000 complete [ 1855.462188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1855.498194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1855.516965] Lustre: Skipped 1 previous similar message [ 1859.358711] Lustre: server umount lustre-MDT0000 complete [ 1860.580301] LustreError: 53325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1860.603698] LustreError: 53325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1863.582688] LustreError: 53310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587646 with bad export cookie 14184677473527803195 [ 1863.603758] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1864.055885] Lustre: server umount lustre-MDT0001 complete [ 1877.630391] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 1880.823508] Key type lgssc unregistered [ 1881.216132] LNet: 56868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1881.242212] LNetError: 56868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.280501] LNet: Removed LNI 192.168.202.109@tcp [ 1883.419714] Key type .llcrypt unregistered [ 1883.423770] Key type ._llcrypt unregistered [ 1912.847952] Key type ._llcrypt registered [ 1912.849688] Key type .llcrypt registered [ 1912.959900] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 1914.110408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1914.243042] alg: No test for adler32 (adler32-zlib) [ 1915.260619] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 1915.446880] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 1917.088223] Key type lgssc registered [ 1918.203843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1925.705385] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1925.732307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1941.218990] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1946.337272] LustreError: 58250:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 1973.088707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.881922] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 1988.611844] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1997.448428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1998.202564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1998.251253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2002.905902] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2003.444821] LustreError: 58275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2003.492119] LustreError: 58275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2007.535719] LustreError: 59287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2007.567726] LustreError: 59287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2011.947527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2012.361139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2017.389737] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2025.542875] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2032.561949] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2038.242815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2038.263556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2038.264913] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2038.280877] Lustre: Skipped 1 previous similar message [ 2043.362083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2043.371921] Lustre: Skipped 1 previous similar message [ 2048.488944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2048.497798] Lustre: Skipped 1 previous similar message [ 2048.992564] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2049.383064] Lustre: server umount lustre-OST0000 complete [ 2073.574599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2073.598591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2073.613339] Lustre: Skipped 1 previous similar message [ 2073.617917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2078.178336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2078.189930] Lustre: Skipped 3 previous similar messages [ 2078.949727] Lustre: server umount lustre-MDT0000 complete [ 2082.369554] LustreError: 58256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788587865 with bad export cookie 10953351764417552288 [ 2082.371786] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2082.378512] LustreError: 58256:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2088.717923] Lustre: server umount lustre-MDT0001 complete [ 2101.406869] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2104.461837] Key type lgssc unregistered [ 2104.883522] LNet: 61999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2104.889486] LNetError: 61999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2104.911442] LNet: Removed LNI 192.168.202.109@tcp [ 2105.942934] Key type .llcrypt unregistered [ 2105.950914] Key type ._llcrypt unregistered [ 2134.727390] Key type ._llcrypt registered [ 2134.731850] Key type .llcrypt registered [ 2134.977394] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2136.490173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2136.703750] alg: No test for adler32 (adler32-zlib) [ 2137.864718] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 2138.214200] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2139.928192] Key type lgssc registered [ 2141.459398] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2151.089454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2151.119087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2152.781835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2158.497479] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2162.148249] LustreError: 63411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.175702] LustreError: 63411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2167.265902] LustreError: 63410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.835369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.241090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2173.277520] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2182.427446] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2189.119525] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2196.025831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2196.412303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2197.447444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2203.183175] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2212.060528] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2233.315390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2233.327080] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2233.340907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2234.346853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2234.350104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2234.380616] Lustre: Skipped 1 previous similar message [ 2236.005681] Lustre: server umount lustre-OST0000 complete [ 2239.969569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2239.978626] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2239.998732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2244.577739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2244.585103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2246.220491] Lustre: server umount lustre-MDT0000 complete [ 2249.700388] LustreError: 63405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2249.726047] LustreError: 63405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2250.016439] LustreError: 63392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588033 with bad export cookie 5321097238703610391 [ 2250.031856] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2250.370538] Lustre: server umount lustre-MDT0001 complete [ 2265.180607] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2268.530895] Key type lgssc unregistered [ 2268.904157] LNet: 66963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2268.916374] LNetError: 66963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2268.952048] LNet: Removed LNI 192.168.202.109@tcp [ 2270.323799] Key type .llcrypt unregistered [ 2270.327847] Key type ._llcrypt unregistered [ 2301.854776] Key type ._llcrypt registered [ 2301.862979] Key type .llcrypt registered [ 2302.006745] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2303.637247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2303.671187] alg: No test for adler32 (adler32-zlib) [ 2304.901241] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 2305.115388] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2306.816818] Key type lgssc registered [ 2308.235550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2317.680167] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 02:01:39 (1788588099) [ 2324.641309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2324.666517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.153757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2330.589587] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2336.748235] LustreError: 68677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2336.766209] LustreError: 68677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2337.768593] LustreError: 68676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2339.939594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2340.344133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2345.930531] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2355.074875] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2361.152884] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2368.635540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2369.063532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2372.191277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2378.782241] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2389.060314] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2395.791448] Lustre: Failing over lustre-MDT0000 [ 2395.937306] Lustre: server umount lustre-MDT0000 complete [ 2396.645268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2396.651357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.678254] LustreError: 68676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2400.744847] LustreError: 68677:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2400.753663] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2400.766284] LustreError: 68677:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2400.800370] Lustre: Skipped 1 previous similar message [ 2405.060690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2405.238462] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2405.426551] Lustre: *** cfs_fail_loc=135, val=0*** [ 2405.511662] LustreError: 71765:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2405.522808] LustreError: 71765:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.109@tcp: cfg command failed: rc = -2 [ 2405.532827] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2405.532827] [ 2405.554426] LustreError: MGC192.168.202.109@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2405.583530] LustreError: 71754:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2405.592136] LustreError: 71754:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 2405.606657] LustreError: 71754:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2405.611299] LustreError: 71754:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2405.629844] Lustre: server umount lustre-MDT0000 complete [ 2405.636480] LustreError: 71754:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2405.864584] LustreError: 69404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2405.900411] LustreError: 69404:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2413.057376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2413.248801] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2413.268273] LustreError: 68668:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2413.272648] LustreError: 72111:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.202.109@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000549e79f0) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2413.614658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2413.655713] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2417.556668] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2418.491706] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2418.697695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2418.762782] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2418.813594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2428.917209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.935603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2428.951869] Lustre: Skipped 1 previous similar message [ 2428.976950] Lustre: Skipped 1 previous similar message [ 2434.019071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2434.262332] Lustre: server umount lustre-OST0000 complete [ 2439.139229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2439.161183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2439.170933] Lustre: Skipped 2 previous similar messages [ 2444.260865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2444.268435] Lustre: Skipped 1 previous similar message [ 2444.822080] Lustre: server umount lustre-MDT0000 complete [ 2448.854836] LustreError: 68658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588232 with bad export cookie 6475492670949406293 [ 2448.873147] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2449.180185] Lustre: server umount lustre-MDT0001 complete [ 2461.601551] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2464.121974] Key type lgssc unregistered [ 2464.362795] LNet: 73549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2464.368542] LNetError: 73549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2465.451232] LNet: Removed LNI 192.168.202.109@tcp [ 2466.386407] Key type .llcrypt unregistered [ 2466.389050] Key type ._llcrypt unregistered [ 2490.149737] Key type ._llcrypt registered [ 2490.152292] Key type .llcrypt registered [ 2490.275279] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2491.599926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2491.669416] alg: No test for adler32 (adler32-zlib) [ 2492.718680] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 2492.968163] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2494.672970] Key type lgssc registered [ 2495.769860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2503.379756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2503.407073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2504.764888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2509.386411] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2513.903164] LustreError: 74957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2513.940312] LustreError: 74957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2518.263820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2518.614075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2523.822691] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2531.448205] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2538.053904] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2545.344036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2545.748120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2548.850863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2552.750838] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2560.757068] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2570.119507] Lustre: Failing over lustre-MDT0000 [ 2570.395962] Lustre: server umount lustre-MDT0000 complete [ 2573.263940] LustreError: 74953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2573.811559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2573.837705] Lustre: Skipped 2 previous similar messages [ 2578.329155] LustreError: 74952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.356316] LustreError: 74952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2581.279975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2581.454115] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2581.667514] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2581.818563] LustreError: 78043:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2581.827132] LustreError: 78043:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.109@tcp: cfg command failed: rc = -9 [ 2581.839470] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2581.839470] [ 2581.855054] LustreError: MGC192.168.202.109@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2581.878086] LustreError: 78032:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2581.885510] LustreError: 78032:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -9 [ 2581.919239] LustreError: 78032:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2581.922492] LustreError: 78032:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2581.987308] Lustre: server umount lustre-MDT0000 complete [ 2581.992610] LustreError: 78032:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2583.459376] LustreError: 74952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2583.496896] LustreError: 74952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2588.563706] LustreError: 74951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2588.584609] LustreError: 74951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2589.836931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.909215] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2589.920597] LustreError: 74949:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2590.187169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2590.260938] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2593.680680] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2595.322681] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2595.366736] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2595.437184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2595.804942] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2604.517633] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2604.529086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.558075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2605.542109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2605.549107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2610.285215] Lustre: server umount lustre-OST0000 complete [ 2615.778950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.787222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2615.793323] Lustre: Skipped 1 previous similar message [ 2615.801573] Lustre: Skipped 1 previous similar message [ 2620.042527] Lustre: server umount lustre-MDT0000 complete [ 2620.903613] LustreError: 74957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2620.916488] LustreError: 74957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2623.958043] LustreError: 74938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588407 with bad export cookie 4215394604217434119 [ 2623.979794] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2623.982537] LustreError: 74938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2624.390600] Lustre: server umount lustre-MDT0001 complete [ 2638.338974] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2641.334922] Key type lgssc unregistered [ 2641.629881] LNet: 79823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2641.640587] LNetError: 79823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2642.661862] LNet: Removed LNI 192.168.202.109@tcp [ 2643.615952] Key type .llcrypt unregistered [ 2643.618753] Key type ._llcrypt unregistered [ 2658.042279] Key type ._llcrypt registered [ 2658.044386] Key type .llcrypt registered [ 2658.227380] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2658.543517] Key type .llcrypt unregistered [ 2658.545353] Key type ._llcrypt unregistered [ 2684.835254] Key type ._llcrypt registered [ 2684.840441] Key type .llcrypt registered [ 2685.025498] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2686.254378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2686.339660] alg: No test for adler32 (adler32-zlib) [ 2687.765718] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 2688.062727] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2689.832170] Key type lgssc registered [ 2691.928838] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2702.505873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2702.520298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.218728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2708.764223] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2714.601561] LustreError: 81722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2714.626211] LustreError: 81722:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2718.412316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2718.876973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2723.517172] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2731.555670] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2738.372510] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2746.156850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2746.606846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2749.763212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2755.948854] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2767.682847] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2778.598327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2778.602120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2778.614234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2783.510391] Lustre: server umount lustre-OST0000 complete [ 2788.836409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2788.859868] Lustre: Skipped 2 previous similar messages [ 2788.861376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2788.886623] Lustre: Skipped 2 previous similar messages [ 2793.954338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2794.336240] Lustre: server umount lustre-MDT0000 complete [ 2798.053598] LustreError: 84084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2798.077145] LustreError: 84084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2799.355731] LustreError: 81703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588582 with bad export cookie 307973886383257409 [ 2799.364483] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2799.379829] LustreError: 81703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2799.877983] Lustre: server umount lustre-MDT0001 complete [ 2811.915450] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2814.834655] Key type lgssc unregistered [ 2815.133631] LNet: 85259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2815.137354] LNetError: 85259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2815.150413] LNet: Removed LNI 192.168.202.109@tcp [ 2815.965720] Key type .llcrypt unregistered [ 2815.967746] Key type ._llcrypt unregistered [ 2838.048720] Key type ._llcrypt registered [ 2838.051838] Key type .llcrypt registered [ 2838.155667] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2839.369061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2839.419549] alg: No test for adler32 (adler32-zlib) [ 2840.463048] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 2840.716993] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2842.384184] Key type lgssc registered [ 2843.565796] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2851.075341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2851.095092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2852.607959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2857.161850] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2862.080951] LustreError: 86668:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2862.112733] LustreError: 86668:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2867.173551] LustreError: 86669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.259490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.743557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2873.068622] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2882.002433] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2888.851346] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2897.648455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2898.107523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2905.922320] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 2918.021495] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2922.733758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2933.220461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2933.226229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2933.248707] Lustre: Skipped 1 previous similar message [ 2937.576117] Lustre: server umount lustre-OST0000 complete [ 2943.460317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.463681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2943.469844] Lustre: Skipped 1 previous similar message [ 2943.477711] Lustre: Skipped 2 previous similar messages [ 2948.577697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2948.889192] Lustre: server umount lustre-MDT0000 complete [ 2952.535830] LustreError: 86649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588735 with bad export cookie 4448191474385783650 [ 2952.553333] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2953.047426] Lustre: server umount lustre-MDT0001 complete [ 2965.332929] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 2968.028248] Key type lgssc unregistered [ 2968.333249] LNet: 90223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2968.343489] LNetError: 90223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2968.356690] LNet: Removed LNI 192.168.202.109@tcp [ 2969.279986] Key type .llcrypt unregistered [ 2969.286742] Key type ._llcrypt unregistered [ 2994.738377] Key type ._llcrypt registered [ 2994.740604] Key type .llcrypt registered [ 2994.859943] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 2996.324980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2996.428238] alg: No test for adler32 (adler32-zlib) [ 2997.556369] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 2997.767502] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 2999.464892] Key type lgssc registered [ 3000.680610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3009.247123] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3009.292517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3010.877098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3015.338733] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3020.261056] LustreError: 91630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3020.281091] LustreError: 91630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3024.258688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3024.554858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3029.491771] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3037.799366] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3044.296239] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3051.747184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3052.065438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3055.184532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3059.189468] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3067.029503] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3080.673596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3080.684341] LustreError: Skipped 1 previous similar message [ 3080.684910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.706658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3084.770817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3084.782705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3084.789617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3084.809819] Lustre: Skipped 1 previous similar message [ 3086.733328] Lustre: server umount lustre-OST0000 complete [ 3091.425333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3091.440403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3091.465415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3095.018607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3095.024432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3097.403970] Lustre: server umount lustre-MDT0000 complete [ 3100.134689] LustreError: 91631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.171670] LustreError: 91631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3101.411360] LustreError: 91613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788588884 with bad export cookie 6897269547528140659 [ 3101.423376] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3101.448623] LustreError: 91613:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3102.111748] Lustre: server umount lustre-MDT0001 complete [ 3115.733400] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 3119.437773] Key type lgssc unregistered [ 3119.838984] LNet: 95181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3119.847476] LNetError: 95181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3119.869981] LNet: Removed LNI 192.168.202.109@tcp [ 3120.882812] Key type .llcrypt unregistered [ 3120.891751] Key type ._llcrypt unregistered [ 3149.500732] Key type ._llcrypt registered [ 3149.504727] Key type .llcrypt registered [ 3149.665795] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3151.399705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3151.602247] alg: No test for adler32 (adler32-zlib) [ 3152.904680] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 3153.316317] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 3155.040523] Key type lgssc registered [ 3156.901564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3165.189370] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3165.223166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3181.092376] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3186.212904] LustreError: 96575:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 3212.896262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3219.031140] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3227.024588] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3233.240880] Lustre: server umount lustre-OST0000 complete [ 3241.549081] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 02:17:03 (1788589023) [ 3248.367443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3248.915452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3253.060903] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3254.252956] LustreError: 98214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3254.276659] LustreError: 98214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3259.363502] LustreError: 98215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3262.180666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3262.535717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3266.949499] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3274.373672] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3280.263208] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3288.664778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3289.210323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3294.327497] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3296.070902] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3304.061115] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3320.289557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3320.299872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3320.317544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3323.361255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3323.363680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3323.366264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3325.114433] Lustre: server umount lustre-OST0000 complete [ 3333.605772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3333.611045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3333.624231] Lustre: Skipped 1 previous similar message [ 3333.637125] Lustre: Skipped 2 previous similar messages [ 3335.514977] Lustre: server umount lustre-MDT0000 complete [ 3338.723377] LustreError: 98209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3338.749461] LustreError: 98209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3339.269114] LustreError: 98195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589122 with bad export cookie 6726163367169613592 [ 3339.282088] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3339.820590] Lustre: server umount lustre-MDT0001 complete [ 3352.921123] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 3356.006340] Key type lgssc unregistered [ 3356.418968] LNet: 101908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3356.427137] LNetError: 101908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3356.457138] LNet: Removed LNI 192.168.202.109@tcp [ 3357.407208] Key type .llcrypt unregistered [ 3357.412629] Key type ._llcrypt unregistered [ 3401.501233] Key type ._llcrypt registered [ 3401.505898] Key type .llcrypt registered [ 3401.697715] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3403.026289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3403.151905] alg: No test for adler32 (adler32-zlib) [ 3404.248277] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 3404.629740] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 3406.368184] Key type lgssc registered [ 3407.810671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3415.731562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3415.751543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3417.209485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.628991] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3427.303610] LustreError: 104090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3427.335281] LustreError: 104090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3431.015075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3431.309453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3436.166510] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3444.851499] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3451.416425] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3459.494366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3459.931120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3462.021053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3466.519453] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3475.114369] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3486.691935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3486.693992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3486.695621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3486.704148] Lustre: Skipped 1 previous similar message [ 3487.928633] Lustre: server umount lustre-OST0000 complete [ 3492.833749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3492.844528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3492.858093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3492.867171] Lustre: Skipped 1 previous similar message [ 3496.931404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3496.945132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3498.349212] Lustre: server umount lustre-MDT0000 complete [ 3502.051637] LustreError: 104090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3502.075404] LustreError: 104090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3502.952478] LustreError: 104071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589286 with bad export cookie 15734418148617104458 [ 3502.959508] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3502.980509] LustreError: 104071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3503.476943] Lustre: server umount lustre-MDT0001 complete [ 3517.178601] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 3520.933523] Key type lgssc unregistered [ 3521.332967] LNet: 107623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3521.348279] LNetError: 107623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3521.368289] LNet: Removed LNI 192.168.202.109@tcp [ 3522.260751] Key type .llcrypt unregistered [ 3522.264170] Key type ._llcrypt unregistered [ 3543.913706] Key type ._llcrypt registered [ 3543.915934] Key type .llcrypt registered [ 3543.993600] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3545.853272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3545.919712] alg: No test for adler32 (adler32-zlib) [ 3547.103881] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 3547.509607] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 3549.248540] Key type lgssc registered [ 3551.211848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3559.738396] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3559.757028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3575.520510] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3580.640570] LustreError: 108862:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 3607.328218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3614.155322] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3623.183318] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3630.263617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3630.574972] LustreError: MGC192.168.202.109@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3630.592683] LustreError: 109888:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3630.604367] LustreError: 109888:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 3630.612713] LustreError: 109888:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3630.704452] Lustre: server umount lustre-MDT0000 complete [ 3630.707923] LustreError: 109888:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3651.552235] Lustre: 108198:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788589418/real 1788589418] req@000000000e2b0ddb x1875471845297536/t0(0) o400->MGC192.168.202.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788589434 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3651.578982] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3651.598932] LustreError: 108874:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3672.207570] Lustre: server umount lustre-OST0000 complete [ 3682.725523] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 3695.166379] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3740.230514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3740.465847] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3740.508211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3740.596242] Lustre: lustre-MDT0000: new disk, initializing [ 3740.673839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3740.707849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3745.732431] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3757.727057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3757.911979] Lustre: 112684:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3757.978125] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3757.981995] Lustre: Skipped 1 previous similar message [ 3758.047485] Lustre: lustre-MDT0001: new disk, initializing [ 3758.122943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3758.160395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3758.174507] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3762.995460] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3773.540574] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3780.567933] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3789.007952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3789.329124] Lustre: lustre-OST0000: new disk, initializing [ 3789.337345] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3789.357286] Lustre: 114653:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3789.540260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.643393] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3798.651856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3798.665189] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3798.774259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3806.928982] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3812.401503] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3812.668460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3817.458750] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3817.730706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3824.609878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3824.614419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3824.617112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3824.646918] Lustre: Skipped 1 previous similar message [ 3829.739243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.761541] Lustre: Skipped 2 previous similar messages [ 3834.336259] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3834.504510] Lustre: server umount lustre-OST0000 complete [ 3839.974852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3839.975273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3839.984567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3840.015430] Lustre: Skipped 1 previous similar message [ 3844.923747] Lustre: server umount lustre-MDT0000 complete [ 3849.143087] LustreError: 112678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589632 with bad export cookie 2545804830406486015 [ 3849.146773] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3849.160712] LustreError: 112678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3849.531447] Lustre: server umount lustre-MDT0001 complete [ 3859.152769] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 02:27:21 (1788589641) [ 3863.443831] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3881.651461] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 3893.448485] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 3932.759735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3933.200038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3933.244312] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3933.330534] Lustre: lustre-MDT0000: new disk, initializing [ 3933.413371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3933.444457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3938.992495] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3951.889358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3952.043385] Lustre: 119556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3952.077194] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3952.079982] Lustre: Skipped 1 previous similar message [ 3952.156922] Lustre: lustre-MDT0001: new disk, initializing [ 3952.295865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3952.350115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3952.360824] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3957.210552] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3968.136720] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3976.397994] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3986.591554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3987.004349] Lustre: lustre-OST0000: new disk, initializing [ 3987.012115] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3987.026780] Lustre: 121528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3987.165034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3995.150698] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 3996.764839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3996.776038] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3997.007408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4005.548720] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4011.048557] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4011.312219] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4015.945322] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4016.153802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4022.761791] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4022.764340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4022.766376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4022.766384] Lustre: Skipped 1 previous similar message [ 4022.794278] Lustre: Skipped 1 previous similar message [ 4022.828846] LustreError: Skipped 1 previous similar message [ 4024.510323] Lustre: server umount lustre-OST0000 complete [ 4029.416585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4029.432108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4029.464015] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4029.478895] Lustre: Skipped 1 previous similar message [ 4035.422374] Lustre: server umount lustre-MDT0000 complete [ 4038.118828] LustreError: 121536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4038.152388] LustreError: 121536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4040.103303] LustreError: 120352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589823 with bad export cookie 2545804830406486897 [ 4040.110490] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4040.113428] LustreError: 120352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4040.641900] Lustre: server umount lustre-MDT0001 complete [ 4051.160728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4051.704173] LustreError: 123743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4051.807905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4056.830154] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4057.058159] LustreError: 123744:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4061.167048] LustreError: 123743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4065.821117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4070.831807] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4079.199227] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4086.140827] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4093.319537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4100.768260] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4109.976945] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4117.400491] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4122.594736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4122.602540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4122.614746] Lustre: Skipped 1 previous similar message [ 4122.619439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4122.626102] Lustre: Skipped 2 previous similar messages [ 4128.642428] Lustre: server umount lustre-OST0000 complete [ 4136.938598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4136.957799] Lustre: Skipped 2 previous similar messages [ 4139.956195] Lustre: server umount lustre-MDT0000 complete [ 4142.060901] LustreError: 123743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4142.084383] LustreError: 123743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4145.986883] LustreError: 123725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788589929 with bad export cookie 2545804830406487779 [ 4146.000901] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4146.034677] LustreError: 123725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4146.616624] Lustre: server umount lustre-MDT0001 complete [ 4161.156505] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 4163.955374] Key type lgssc unregistered [ 4164.271372] LNet: 127375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4164.275811] LNetError: 127375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4164.310935] LNet: Removed LNI 192.168.202.109@tcp [ 4165.256796] Key type .llcrypt unregistered [ 4165.258866] Key type ._llcrypt unregistered [ 4185.307912] Key type ._llcrypt registered [ 4185.315809] Key type .llcrypt registered [ 4185.483958] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 4206.715022] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 4207.612155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4207.737074] alg: No test for adler32 (adler32-zlib) [ 4209.036440] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 4209.272815] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 4211.072387] Key type lgssc registered [ 4212.553491] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4256.397757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4256.461636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4257.905557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4257.937539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4258.045645] Lustre: lustre-MDT0000: new disk, initializing [ 4258.111591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4258.127851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4261.550796] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4275.021409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4275.199770] Lustre: 130526:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4275.244639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4275.248243] Lustre: Skipped 1 previous similar message [ 4275.303484] Lustre: lustre-MDT0001: new disk, initializing [ 4275.407336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4275.440793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4275.458353] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4279.979848] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4289.792486] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4295.185888] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4302.159463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4302.417920] Lustre: lustre-OST0000: new disk, initializing [ 4302.422089] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4302.430882] Lustre: 132494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4302.523761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4307.820285] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4311.628229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4311.637682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4311.704496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4316.512221] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4321.628561] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4321.878962] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4327.100744] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4327.309916] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4332.522073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4332.536345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4332.552836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4332.556851] LustreError: Skipped 1 previous similar message [ 4332.566855] Lustre: Skipped 1 previous similar message [ 4335.700516] Lustre: server umount lustre-OST0000 complete [ 4342.244673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4342.252665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4342.265837] Lustre: Skipped 1 previous similar message [ 4342.281607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4345.859466] Lustre: server umount lustre-MDT0000 complete [ 4350.012613] LustreError: 130519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590133 with bad export cookie 804791201232119800 [ 4350.018705] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4350.025680] LustreError: 130519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4350.501477] Lustre: server umount lustre-MDT0001 complete [ 4358.842603] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 02:35:41 (1788590141) [ 4365.029721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4365.526827] LustreError: 134918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4365.564041] LustreError: 134918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4365.634206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4370.496965] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4370.918695] LustreError: 134919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4375.016853] LustreError: 134918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4379.240349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4379.709906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4383.886568] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4392.031676] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4398.000146] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4400.110627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4400.112450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4400.115070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4400.115079] Lustre: Skipped 1 previous similar message [ 4400.142956] Lustre: Skipped 2 previous similar messages [ 4404.258662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4404.272349] Lustre: Skipped 2 previous similar messages [ 4406.014327] Lustre: server umount lustre-MDT0000 complete [ 4409.314852] LustreError: 134913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4409.332301] LustreError: 134913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4409.418571] LustreError: 134899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590192 with bad export cookie 804791201232120682 [ 4409.424071] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4409.709409] Lustre: server umount lustre-MDT0001 complete [ 4417.883747] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 02:36:40 (1788590200) [ 4424.477117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4440.099491] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4445.217196] LustreError: 137332:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 4470.944761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4477.190193] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4485.602309] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4487.972735] Lustre: server umount lustre-OST0000 complete [ 4497.337857] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 02:37:59 (1788590279) [ 4505.095557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4505.574752] LustreError: 138777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4505.637567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4509.954471] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4515.810749] LustreError: 138777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4515.837456] LustreError: 138777:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4518.922143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4524.183961] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4532.658151] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4539.903116] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4548.781929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4549.231016] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4549.233941] Lustre: Skipped 1 previous similar message [ 4556.331465] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4565.684332] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4575.214592] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4575.217193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4575.257776] Lustre: Skipped 2 previous similar messages [ 4575.269452] Lustre: Skipped 1 previous similar message [ 4581.125606] Lustre: server umount lustre-MDT0000 complete [ 4585.009064] LustreError: 138758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590368 with bad export cookie 804791201232121347 [ 4585.017276] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4585.036155] LustreError: 138758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4585.428369] Lustre: server umount lustre-MDT0001 complete [ 4600.707796] Lustre: server umount lustre-OST0000 complete [ 4612.753720] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 02:39:54 (1788590394) [ 4621.080171] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4621.593668] LustreError: 142432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4621.649473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4627.594533] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4637.267821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4642.742580] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4653.153314] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4659.673175] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4667.827722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4676.688577] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4687.473488] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4691.728766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4693.950646] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4694.446361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4700.506262] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4700.753403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4703.201586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4703.213794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4703.242189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4703.250078] Lustre: Skipped 4 previous similar messages [ 4707.297482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4709.156815] Lustre: server umount lustre-OST0000 complete [ 4714.466491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4714.475858] Lustre: Skipped 1 previous similar message [ 4719.563296] Lustre: server umount lustre-MDT0000 complete [ 4722.657858] LustreError: 143419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4722.679893] LustreError: 143419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4723.646250] LustreError: 142415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590506 with bad export cookie 804791201232122418 [ 4723.651145] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4723.659788] LustreError: 142415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4724.189735] Lustre: server umount lustre-MDT0001 complete [ 4733.777884] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 02:41:55 (1788590515) [ 4741.351550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4756.963958] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4762.080876] LustreError: 146596:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 4787.809056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4787.820728] Lustre: Skipped 2 previous similar messages [ 4794.830203] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4803.024532] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4810.098714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4810.515254] LustreError: 146621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4810.627490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4815.360116] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4824.233273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4829.113866] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4836.969176] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4843.493891] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4848.511612] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4848.808343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4854.230715] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4854.552404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4860.390607] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4860.391605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4860.402020] LustreError: Skipped 1 previous similar message [ 4860.402360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4860.402364] Lustre: Skipped 5 previous similar messages [ 4860.432156] Lustre: Skipped 3 previous similar messages [ 4862.839516] Lustre: server umount lustre-MDT0000 complete [ 4866.962316] LustreError: 148374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590650 with bad export cookie 804791201232123440 [ 4866.966068] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4866.976592] LustreError: 148374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4867.312453] Lustre: server umount lustre-MDT0001 complete [ 4881.773412] Lustre: server umount lustre-OST0000 complete [ 4890.902437] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 02:44:32 (1788590672) [ 4897.628767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4913.184840] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4918.305150] LustreError: 150791:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 4950.134798] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4958.821722] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4966.520163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4967.018739] LustreError: 150816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4967.048912] LustreError: 150816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4967.219603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4972.178804] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4982.067643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4987.120888] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 4996.016515] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5002.772443] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5010.544511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5010.721369] Lustre: lustre-OST0001: new disk, initializing [ 5010.724993] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5010.731388] Lustre: 153665:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5017.455291] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5018.683131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5018.690168] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5018.904440] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5028.035461] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5033.761070] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5034.012663] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5039.443539] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5039.617697] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5044.199812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5044.213568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5044.213669] LustreError: Skipped 1 previous similar message [ 5044.216885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5044.216894] Lustre: Skipped 2 previous similar messages [ 5044.256046] Lustre: Skipped 1 previous similar message [ 5047.588860] Lustre: server umount lustre-OST0000 complete [ 5058.503584] Lustre: server umount lustre-OST0001 complete [ 5068.977817] Lustre: server umount lustre-MDT0000 complete [ 5072.895037] LustreError: 150798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788590856 with bad export cookie 804791201232124392 [ 5072.903177] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5072.905825] LustreError: 150798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5073.286033] Lustre: server umount lustre-MDT0001 complete [ 5097.101911] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 02:47:59 (1788590879) [ 5099.083362] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5101.361811] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5103.058345] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 02:48:05 (1788590885) [ 5109.266910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5109.461135] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5109.482464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5109.613628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5109.621095] Lustre: Skipped 6 previous similar messages [ 5113.856356] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5123.157413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5123.261477] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5123.296803] Lustre: 157226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5128.523116] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5136.302567] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5142.629860] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5149.230923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5149.440217] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5155.974261] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5155.986444] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5164.039443] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5168.819569] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5168.999448] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5173.627549] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5173.814848] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5176.290083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5176.300968] LustreError: Skipped 3 previous similar messages [ 5176.304683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5176.321433] Lustre: Skipped 4 previous similar messages [ 5176.326935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5176.333900] Lustre: Skipped 6 previous similar messages [ 5190.112936] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5190.255290] Lustre: server umount lustre-OST0000 complete [ 5213.014710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5217.514661] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.539267] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5217.541337] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5217.569536] Lustre: Skipped 1 previous similar message [ 5219.622577] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5228.059439] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5233.574504] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5233.749571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5238.744604] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5238.956553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5244.720647] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9d0e87210800.ost_server_uuid 50 [ 5246.649580] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9d0e87210800.ost_server_uuid in FULL state after 0 sec [ 5246.960180] LustreError: 162202:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.202.9@tcp) returned error from blocking AST (req@00000000fccbd4f8 x1875472539717120 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000aab611c6/0xb2b315563cd0342 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.9@tcp remote: 0xf0972488e4a02af7 expref: 5 pid: 157234 timeout: 5346 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5247.035470] LustreError: lustre-MDT0000: A client on nid 192.168.202.9@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5247.062500] LustreError: 157222:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 1s: evicting client at 192.168.202.9@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000aab611c6/0xb2b315563cd0342 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.202.9@tcp remote: 0xf0972488e4a02af7 expref: 6 pid: 157234 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5253.109686] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5253.131173] LustreError: Skipped 1 previous similar message [ 5258.683477] Lustre: server umount lustre-OST0000 complete [ 5268.522123] Lustre: server umount lustre-MDT0000 complete [ 5268.962665] LustreError: 157233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5268.984467] LustreError: 157233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 5272.248401] LustreError: 157217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591055 with bad export cookie 804791201232125575 [ 5272.253629] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5272.259919] LustreError: 157217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5278.717607] Lustre: server umount lustre-MDT0001 complete [ 5290.926646] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 5294.883716] Key type lgssc unregistered [ 5295.253788] LNet: 163340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5295.261708] LNetError: 163340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5295.274040] LNet: Removed LNI 192.168.202.109@tcp [ 5296.459726] Key type .llcrypt unregistered [ 5296.466642] Key type ._llcrypt unregistered [ 5326.286207] Key type ._llcrypt registered [ 5326.289956] Key type .llcrypt registered [ 5326.407962] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5327.307359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5327.427875] alg: No test for adler32 (adler32-zlib) [ 5328.467791] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 5328.655902] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 5330.330246] Key type lgssc registered [ 5331.911716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5341.040434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5341.065388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5342.600501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5347.609727] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5351.913417] LustreError: 164751:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5351.930148] LustreError: 164751:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5356.930052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5357.026976] LustreError: 164752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5357.232275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5361.435269] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5369.163473] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5375.597988] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5381.956415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5382.120565] LustreError: 166572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5382.347832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5387.518465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5388.597912] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5396.744872] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5404.269524] Lustre: Failing over lustre-MDT0000 [ 5404.470532] Lustre: server umount lustre-MDT0000 complete [ 5404.587213] LustreError: 164746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5408.225555] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5408.230558] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5413.348953] LustreError: 164748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5413.383955] LustreError: 164748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5417.051057] LustreError: 164731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.9@tcp arrived at 1788591200 with bad export cookie 13123796144150431035 [ 5417.083851] LustreError: 164731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5423.597038] LustreError: 166870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5423.652663] LustreError: 166870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5424.545230] Lustre: 164062:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788591191/real 1788591191] req@00000000f438fbb5 x1875473712833536/t0(0) o400->MGC192.168.202.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788591207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5424.597483] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5425.383300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5434.851155] LustreError: 164059:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000010c908e9 x1875473712838912/t0(0) o250->MGC192.168.202.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5435.307580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5435.395125] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5440.326363] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5440.489766] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5440.513873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5443.138049] Lustre: lustre-MDT0000: Denying connection for new client 4f4abada-647b-45b0-8d24-7f1f82cd5b63 (at 192.168.202.9@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5454.316145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5454.322349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5454.357976] Lustre: Skipped 2 previous similar messages [ 5454.377643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5455.851498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5455.852928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5460.983127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5460.993798] Lustre: Skipped 1 previous similar message [ 5466.080360] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5466.085929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5466.093385] Lustre: Skipped 2 previous similar messages [ 5466.280276] Lustre: server umount lustre-OST0000 complete [ 5471.172581] LustreError: 168485:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5471.204690] Lustre: 167833:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5471.220424] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5471.223803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5471.274445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5471.284325] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5471.306665] LustreError: 167833:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000064ffb42a x1875473712856576/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5471.332833] LustreError: 167833:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5471.345152] LustreError: 167833:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5471.354639] Lustre: lustre-MDT0000: Recovery over after 0:31, of 2 clients 0 recovered and 2 were evicted. [ 5471.771590] Lustre: server umount lustre-MDT0000 complete [ 5476.153919] LustreError: 164733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591259 with bad export cookie 13123796144150431133 [ 5476.156042] LustreError: 164752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5476.156684] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5476.168155] LustreError: 164733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5476.205168] LustreError: 164752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5478.669211] Lustre: server umount lustre-MDT0001 complete [ 5492.360332] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 5495.826870] Key type lgssc unregistered [ 5496.088078] LNet: 169236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5496.097483] LNetError: 169236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5496.112334] LNet: Removed LNI 192.168.202.109@tcp [ 5497.180888] Key type .llcrypt unregistered [ 5497.182376] Key type ._llcrypt unregistered [ 5524.761893] Key type ._llcrypt registered [ 5524.762968] Key type .llcrypt registered [ 5524.876476] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5526.478722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5526.525496] alg: No test for adler32 (adler32-zlib) [ 5527.559823] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 5527.838779] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 5529.536438] Key type lgssc registered [ 5530.573650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5539.123200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5539.145071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5540.514543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5543.973579] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5550.060114] LustreError: 170645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5550.075248] LustreError: 170645:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5553.039586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5553.406273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5558.800650] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5567.303242] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5574.690502] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5582.942497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5583.376580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5585.398342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5590.606403] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5599.972315] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5608.936506] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5608.939576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5608.956042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5608.978351] Lustre: Skipped 1 previous similar message [ 5608.986606] Lustre: Skipped 1 previous similar message [ 5614.065326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5614.079028] Lustre: Skipped 1 previous similar message [ 5615.263720] Lustre: server umount lustre-OST0000 complete [ 5620.193411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5620.207898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5620.232185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5624.292229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5624.293419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5624.329219] Lustre: Skipped 1 previous similar message [ 5626.713871] Lustre: server umount lustre-MDT0000 complete [ 5629.420471] LustreError: 171373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5629.448381] LustreError: 171373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5631.264194] LustreError: 170626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591414 with bad export cookie 12660532619364319030 [ 5631.277019] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5631.279541] LustreError: 170626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5631.645754] Lustre: server umount lustre-MDT0001 complete [ 5643.984375] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 5646.809062] Key type lgssc unregistered [ 5647.185414] LNet: 174180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5647.198622] LNetError: 174180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5647.222079] LNet: Removed LNI 192.168.202.109@tcp [ 5648.194767] Key type .llcrypt unregistered [ 5648.196536] Key type ._llcrypt unregistered [ 5687.318580] Key type ._llcrypt registered [ 5687.321391] Key type .llcrypt registered [ 5687.479289] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5688.426063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5688.460522] alg: No test for adler32 (adler32-zlib) [ 5689.742174] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 5690.074188] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 5691.728300] Key type lgssc registered [ 5692.966873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5700.405195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5700.437765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5702.028555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5706.286462] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5712.359677] LustreError: 175880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5712.387700] LustreError: 175880:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5714.304668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5714.818308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5719.600831] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5726.192049] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5732.333896] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5739.275808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5739.498352] LustreError: 177700:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5739.746724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5746.290454] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5747.193277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5753.868742] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5762.548877] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5762.557298] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5762.620710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5763.713437] Lustre: server umount lustre-OST0000 complete [ 5767.649513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5767.655155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5767.666202] Lustre: Skipped 2 previous similar messages [ 5767.675805] Lustre: Skipped 2 previous similar messages [ 5772.783924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5772.790983] Lustre: Skipped 1 previous similar message [ 5773.918118] Lustre: server umount lustre-MDT0000 complete [ 5777.737891] LustreError: 175861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591560 with bad export cookie 13032864240701157285 [ 5777.760061] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5777.764312] LustreError: 175861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5778.283793] Lustre: server umount lustre-MDT0001 complete [ 5792.112669] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 5795.559624] Key type lgssc unregistered [ 5795.872723] LNet: 179411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5795.880567] LNetError: 179411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5795.899281] LNet: Removed LNI 192.168.202.109@tcp [ 5796.814180] Key type .llcrypt unregistered [ 5796.817851] Key type ._llcrypt unregistered [ 5821.323564] Key type ._llcrypt registered [ 5821.326892] Key type .llcrypt registered [ 5821.430393] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5822.331572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5822.604539] alg: No test for adler32 (adler32-zlib) [ 5823.730949] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 5824.028624] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 5825.696452] Key type lgssc registered [ 5826.996751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5838.733744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5838.784460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5840.252131] Lustre: *** cfs_fail_loc=135, val=0*** [ 5840.415295] LustreError: 180931:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5840.425721] LustreError: 180931:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.109@tcp: cfg command failed: rc = -2 [ 5840.449062] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5840.449062] [ 5840.476873] LustreError: MGC192.168.202.109@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5840.491529] LustreError: 180892:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5840.501721] LustreError: 180892:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 5840.514759] LustreError: 180892:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5840.523749] LustreError: 180892:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5840.604447] Lustre: server umount lustre-MDT0000 complete [ 5840.611860] LustreError: 180892:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5847.499311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5847.859618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5852.872763] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5858.284161] LustreError: 181303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5858.316428] LustreError: 181303:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5863.102929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5863.393721] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5863.579479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5869.076850] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 5877.752810] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5884.803909] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5887.521616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5887.540622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5887.542465] Lustre: Skipped 1 previous similar message [ 5887.582364] Lustre: Skipped 1 previous similar message [ 5888.610763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5888.620222] Lustre: Skipped 1 previous similar message [ 5894.033702] Lustre: server umount lustre-MDT0000 complete [ 5894.120946] LustreError: 181303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5894.165175] LustreError: 181303:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5898.745139] LustreError: 181284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591681 with bad export cookie 1739242771740881493 [ 5898.752448] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5898.760374] LustreError: 181284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5899.150433] Lustre: server umount lustre-MDT0001 complete [ 5911.235220] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 5915.029602] Key type lgssc unregistered [ 5915.333741] LNet: 183636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5915.342630] LNetError: 183636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5915.363087] LNet: Removed LNI 192.168.202.109@tcp [ 5916.374213] Key type .llcrypt unregistered [ 5916.378954] Key type ._llcrypt unregistered [ 5946.853791] Key type ._llcrypt registered [ 5946.856608] Key type .llcrypt registered [ 5946.946060] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 5947.180320] Key type .llcrypt unregistered [ 5947.182607] Key type ._llcrypt unregistered [ 5967.389467] Key type ._llcrypt registered [ 5967.390500] Key type .llcrypt registered [ 5967.462505] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 5968.355274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5968.542669] alg: No test for adler32 (adler32-zlib) [ 5969.636360] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 5970.066887] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 5971.839062] Key type lgssc registered [ 5973.020880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5980.508652] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5980.524046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5996.001174] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6001.120405] LustreError: 185764:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.109@tcp: failed processing log, type 4: rc = -110 [ 6027.872876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6034.431709] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6043.311239] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6050.859940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6051.619201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6051.709763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6056.363227] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6056.938167] LustreError: 185790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6057.007837] LustreError: 185790:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6062.050536] LustreError: 185789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6062.089295] LustreError: 185789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6067.171987] LustreError: 186801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6067.207687] LustreError: 186801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6068.220215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6068.633475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6073.998318] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6083.633297] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6091.317545] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6094.734704] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6104.549696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6104.551726] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6104.592445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6104.606558] Lustre: Skipped 1 previous similar message [ 6109.669599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6109.686581] Lustre: Skipped 1 previous similar message [ 6110.641752] Lustre: server umount lustre-OST0000 complete [ 6119.906898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6119.910333] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6119.912181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6119.937740] Lustre: Skipped 2 previous similar messages [ 6125.027241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6125.032714] Lustre: Skipped 2 previous similar messages [ 6129.120141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6129.363831] Lustre: server umount lustre-MDT0000 complete [ 6130.161949] LustreError: 186807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6130.182597] LustreError: 186807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6133.719118] LustreError: 188425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788591916 with bad export cookie 3698851682680475319 [ 6133.724345] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6133.741124] LustreError: 188425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6134.185445] Lustre: server umount lustre-MDT0001 complete [ 6148.085928] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6150.996440] Key type lgssc unregistered [ 6151.268313] LNet: 189683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6151.277781] LNetError: 189683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6151.301826] LNet: Removed LNI 192.168.202.109@tcp [ 6152.485237] Key type .llcrypt unregistered [ 6152.490150] Key type ._llcrypt unregistered [ 6180.891397] Key type ._llcrypt registered [ 6180.893107] Key type .llcrypt registered [ 6181.002433] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6182.526592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6182.640982] alg: No test for adler32 (adler32-zlib) [ 6183.817821] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 6184.107509] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6185.793280] Key type lgssc registered [ 6187.163410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6196.272709] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6196.317713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6197.968037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6202.778936] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6208.489994] LustreError: 191093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6208.511313] LustreError: 191093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6211.610035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6212.018497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6216.721245] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6225.191594] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6232.192979] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6241.333185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6241.727083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6243.785288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6249.676776] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6260.695129] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6268.456503] Lustre: Failing over lustre-MDT0000 [ 6268.890627] LustreError: 191087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6269.043531] Lustre: server umount lustre-MDT0000 complete [ 6272.489171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6272.496019] LustreError: 191090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6272.501151] Lustre: Skipped 2 previous similar messages [ 6272.541124] LustreError: 191090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6277.608319] LustreError: 191087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6277.632961] LustreError: 191087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6282.724144] LustreError: 191090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6282.750026] LustreError: 191090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6287.841877] Lustre: 190405:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788592055/real 1788592055] req@00000000f38e346b x1875474610418688/t0(0) o400->MGC192.168.202.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788592071 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6287.876390] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6288.551365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6292.969752] LustreError: 192098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6292.997709] LustreError: 192098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6298.479781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6298.542363] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6300.557374] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6303.725726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6303.752169] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6303.822638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6303.968322] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6309.368647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6313.568521] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6328.804100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6328.813749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6328.840745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6332.075850] Lustre: server umount lustre-OST0000 complete [ 6339.559903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6339.574534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6339.595448] Lustre: Skipped 1 previous similar message [ 6339.599144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6339.627643] Lustre: Skipped 3 previous similar messages [ 6343.051835] Lustre: server umount lustre-MDT0000 complete [ 6344.675261] LustreError: 192098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6344.696451] LustreError: 192098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6347.241581] LustreError: 194678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592130 with bad export cookie 11475392308677277631 [ 6347.243288] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6347.250314] LustreError: 194678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6347.606345] Lustre: server umount lustre-MDT0001 complete [ 6360.329152] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6363.109292] Key type lgssc unregistered [ 6363.416680] LNet: 195887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6363.425826] LNetError: 195887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6363.460468] LNet: Removed LNI 192.168.202.109@tcp [ 6364.336937] Key type .llcrypt unregistered [ 6364.338913] Key type ._llcrypt unregistered [ 6389.136389] Key type ._llcrypt registered [ 6389.139831] Key type .llcrypt registered [ 6389.286262] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6390.253179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6390.390091] alg: No test for adler32 (adler32-zlib) [ 6391.530875] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 6391.842574] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6393.520168] Key type lgssc registered [ 6394.552760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6402.061206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6402.079511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6403.489211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6407.679079] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6413.797833] LustreError: 197297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6413.832894] LustreError: 197297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6416.668734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6416.992531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6421.941641] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6429.520614] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6435.587912] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6442.884861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6443.342694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6448.507832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6450.518349] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6459.091881] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6463.379274] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6467.924452] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6486.938251] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6498.803574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6498.804437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6498.837968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6498.855753] Lustre: Skipped 1 previous similar message [ 6502.269823] Lustre: server umount lustre-OST0000 complete [ 6509.027732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6509.037783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6509.072478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6509.074715] Lustre: Skipped 1 previous similar message [ 6514.148113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6514.153568] Lustre: Skipped 1 previous similar message [ 6519.271158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6519.277778] Lustre: Skipped 2 previous similar messages [ 6521.314276] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6521.644789] Lustre: server umount lustre-MDT0000 complete [ 6524.391698] LustreError: 197294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6524.412769] LustreError: 197294:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6526.590381] LustreError: 197277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592309 with bad export cookie 17817389776537871053 [ 6526.600920] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6526.605550] LustreError: 197277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6527.065654] Lustre: server umount lustre-MDT0001 complete [ 6540.206739] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6544.127315] Key type lgssc unregistered [ 6544.473850] LNet: 201168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6544.487601] LNetError: 201168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6544.518803] LNet: Removed LNI 192.168.202.109@tcp [ 6545.940181] Key type .llcrypt unregistered [ 6545.946229] Key type ._llcrypt unregistered [ 6565.098844] Key type ._llcrypt registered [ 6565.101131] Key type .llcrypt registered [ 6565.234762] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6566.279668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6566.305699] alg: No test for adler32 (adler32-zlib) [ 6567.478653] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 6567.651759] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6569.296196] Key type lgssc registered [ 6570.372395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6578.643827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6578.662768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6580.328335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6585.027542] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6590.446185] LustreError: 202432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6590.477250] LustreError: 202432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6594.564835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6594.938864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6600.144952] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6607.826793] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6613.759743] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6620.588333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6621.025562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6625.163155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6628.850818] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6638.397802] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6645.246703] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6654.634134] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6665.311662] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6671.139550] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6680.551144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6680.560759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6680.563251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6685.666741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6685.673510] Lustre: Skipped 2 previous similar messages [ 6690.787880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6690.797518] Lustre: Skipped 1 previous similar message [ 6694.370908] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6694.597263] Lustre: server umount lustre-OST0000 complete [ 6701.032372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6701.049042] Lustre: Skipped 1 previous similar message [ 6701.059856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6701.078218] Lustre: Skipped 1 previous similar message [ 6706.148242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6706.157036] Lustre: Skipped 1 previous similar message [ 6713.312254] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6713.553109] Lustre: server umount lustre-MDT0000 complete [ 6716.388110] LustreError: 202432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6716.415121] LustreError: 202432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6717.479240] LustreError: 205111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592500 with bad export cookie 11771222572974198343 [ 6717.488248] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6717.492353] LustreError: 205111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6718.005614] Lustre: server umount lustre-MDT0001 complete [ 6729.355870] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6732.358767] Key type lgssc unregistered [ 6732.704664] LNet: 207314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6732.715988] LNetError: 207314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6732.733641] LNet: Removed LNI 192.168.202.109@tcp [ 6733.683332] Key type .llcrypt unregistered [ 6733.686716] Key type ._llcrypt unregistered [ 6755.197449] Key type ._llcrypt registered [ 6755.200389] Key type .llcrypt registered [ 6755.295670] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6756.129593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6756.162634] alg: No test for adler32 (adler32-zlib) [ 6757.299136] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 6757.553498] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6759.249041] Key type lgssc registered [ 6760.324292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6768.020784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6768.045783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6769.469315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6774.312993] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6778.851352] LustreError: 208723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6778.870907] LustreError: 208723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6781.955792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6782.256040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6786.697427] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6794.472934] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6801.137640] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6807.421124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6807.880884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6813.997951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6814.240747] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6821.888696] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6837.731973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6837.738763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6837.745043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6837.766552] Lustre: Skipped 1 previous similar message [ 6837.791410] Lustre: Skipped 1 previous similar message [ 6842.851357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6842.869222] Lustre: Skipped 1 previous similar message [ 6843.608850] Lustre: server umount lustre-OST0000 complete [ 6847.971187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6847.983338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6847.998379] Lustre: Skipped 1 previous similar message [ 6848.018571] Lustre: Skipped 1 previous similar message [ 6853.090817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6853.100662] Lustre: Skipped 1 previous similar message [ 6854.057573] Lustre: server umount lustre-MDT0000 complete [ 6858.224273] LustreError: 209452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6858.237376] LustreError: 209452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6858.825162] LustreError: 208705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592641 with bad export cookie 9424100602211360824 [ 6858.836775] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6858.841456] LustreError: 208705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6859.209843] Lustre: server umount lustre-MDT0001 complete [ 6873.396504] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 6876.906291] Key type lgssc unregistered [ 6877.254751] LNet: 212410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6877.267347] LNetError: 212410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6877.294453] LNet: Removed LNI 192.168.202.109@tcp [ 6878.056730] Key type .llcrypt unregistered [ 6878.058776] Key type ._llcrypt unregistered [ 6897.596392] Key type ._llcrypt registered [ 6897.598767] Key type .llcrypt registered [ 6897.767759] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 6898.723491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6898.845995] alg: No test for adler32 (adler32-zlib) [ 6899.949835] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 6900.248581] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 6902.032195] Key type lgssc registered [ 6903.217772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6911.201702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6911.236613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6912.553830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6917.396547] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6922.726085] LustreError: 213676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6922.752909] LustreError: 213676:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6927.849502] LustreError: 213675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6927.931831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6928.500474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6933.445430] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6941.759719] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6948.632796] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6957.547947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6958.159026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6965.109480] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 6973.434570] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6982.724633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7003.107794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7003.126459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7003.129435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7003.153448] Lustre: Skipped 1 previous similar message [ 7003.166702] Lustre: Skipped 1 previous similar message [ 7006.427274] Lustre: server umount lustre-OST0000 complete [ 7013.348543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7013.354268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7013.360878] Lustre: Skipped 1 previous similar message [ 7013.387488] Lustre: Skipped 1 previous similar message [ 7017.352124] Lustre: server umount lustre-MDT0000 complete [ 7018.476886] LustreError: 216338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7020.952803] LustreError: 213655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788592804 with bad export cookie 16480086520463564831 [ 7020.963970] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7020.973208] LustreError: 213655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7021.532960] Lustre: server umount lustre-MDT0001 complete [ 7032.593334] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 7035.382462] Key type lgssc unregistered [ 7035.684614] LNet: 217378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7035.691374] LNetError: 217378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7035.715818] LNet: Removed LNI 192.168.202.109@tcp [ 7036.409750] Key type .llcrypt unregistered [ 7036.412388] Key type ._llcrypt unregistered [ 7054.114714] Key type ._llcrypt registered [ 7054.118784] Key type .llcrypt registered [ 7054.207554] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 7055.166649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7055.187824] alg: No test for adler32 (adler32-zlib) [ 7056.209166] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 7056.408119] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 7058.080262] Key type lgssc registered [ 7059.058766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7066.182612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7066.219650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7067.734976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7071.639770] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7076.842300] LustreError: 218647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7076.862273] LustreError: 218647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7080.218060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7080.760595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7085.878375] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7093.381449] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7100.138293] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7108.008340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7108.282043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7112.373932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7115.884675] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7124.604109] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7135.018670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7135.070882] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7135.268827] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7141.495194] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7151.103738] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7153.563467] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7165.629238] Lustre: Permanently deactivating lustre-OST0001 [ 7165.649114] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7174.787100] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7181.020907] Lustre: Permanently reactivating lustre-OST0001 [ 7181.053786] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7181.056236] Lustre: Skipped 2 previous similar messages [ 7191.145734] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7191.171267] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7191.177757] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7191.189859] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7191.217411] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7194.091741] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7194.118317] Lustre: Skipped 1 previous similar message [ 7194.126187] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7194.137609] Lustre: Skipped 1 previous similar message [ 7198.731540] Lustre: server umount lustre-OST0001 complete [ 7199.227069] LustreError: 220468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7199.259250] LustreError: 220468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7204.322838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7204.324844] LustreError: 221884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7204.333562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7204.336165] Lustre: Skipped 2 previous similar messages [ 7204.375104] LustreError: 221884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7209.330395] Lustre: server umount lustre-OST0000 complete [ 7214.049273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7214.058525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7214.071687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7214.074535] Lustre: Skipped 1 previous similar message [ 7219.276560] Lustre: server umount lustre-MDT0000 complete [ 7219.687324] LustreError: 218647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7219.707398] LustreError: 218647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7223.602781] LustreError: 222440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593006 with bad export cookie 18145027509281818294 [ 7223.614891] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7223.621727] LustreError: 222440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7224.357595] Lustre: server umount lustre-MDT0001 complete [ 7237.708774] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 7240.287375] Key type lgssc unregistered [ 7240.600573] LNet: 223703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7240.625962] LNetError: 223703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7241.639880] LNet: Removed LNI 192.168.202.109@tcp [ 7242.372238] Key type .llcrypt unregistered [ 7242.374875] Key type ._llcrypt unregistered [ 7282.404609] Key type ._llcrypt registered [ 7282.407985] Key type .llcrypt registered [ 7282.517637] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 7283.373261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7283.492937] alg: No test for adler32 (adler32-zlib) [ 7284.495256] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 7284.745610] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 7286.561872] Key type lgssc registered [ 7287.368435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7294.861854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7294.879706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7296.042554] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7296.056757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7296.168318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7300.749916] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7309.728332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7309.811109] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7309.840344] Lustre: 225904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7310.068204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7314.778823] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7322.976278] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7329.354485] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7336.274456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7336.427291] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7336.628731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7343.058732] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7346.289731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7351.146617] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7355.558447] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7361.308295] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7368.139894] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7378.263511] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7386.413389] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7403.230412] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7403.237603] Lustre: Skipped 1 previous similar message [ 7437.777430] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7437.786806] Lustre: Skipped 3 previous similar messages [ 7504.576399] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7504.583774] Lustre: Skipped 7 previous similar messages [ 7545.829706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7545.849141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.892748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7546.348784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7546.361881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7548.167921] Lustre: server umount lustre-OST0000 complete [ 7556.072167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7556.089474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7556.107701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7556.120382] Lustre: Skipped 2 previous similar messages [ 7558.598883] Lustre: server umount lustre-MDT0000 complete [ 7561.703169] LustreError: 228711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7561.711167] LustreError: 228711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7562.955156] LustreError: 227766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593346 with bad export cookie 7881680054860725705 [ 7562.957183] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7562.971169] LustreError: 227766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7563.286253] Lustre: server umount lustre-MDT0001 complete [ 7575.000564] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 7578.047570] Key type lgssc unregistered [ 7578.361371] LNet: 230728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7578.375353] LNetError: 230728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7578.394994] LNet: Removed LNI 192.168.202.109@tcp [ 7579.373280] Key type .llcrypt unregistered [ 7579.377434] Key type ._llcrypt unregistered [ 7607.405673] Key type ._llcrypt registered [ 7607.407045] Key type .llcrypt registered [ 7607.548639] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 7608.753085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7608.938758] alg: No test for adler32 (adler32-zlib) [ 7610.152500] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 7610.465662] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 7612.170493] Key type lgssc registered [ 7613.703816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7623.965715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7624.001382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7625.669113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7630.725735] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7634.922181] LustreError: 232139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7634.958067] LustreError: 232139:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7640.034851] LustreError: 232140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7640.752298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7641.349681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7646.694762] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7655.964916] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7662.986906] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7670.570313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7670.969949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7678.017349] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_default_debug -1 all [ 7687.333137] Lustre: DEBUG MARKER: oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7695.677359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7721.446731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7721.454901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7721.467133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7721.482134] Lustre: Skipped 1 previous similar message [ 7721.806201] Lustre: server umount lustre-OST0000 complete [ 7727.589714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7727.597687] Lustre: Skipped 1 previous similar message [ 7727.601432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7727.606109] Lustre: Skipped 1 previous similar message [ 7732.709125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7732.720272] Lustre: Skipped 1 previous similar message [ 7737.842272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7737.858404] Lustre: Skipped 1 previous similar message [ 7740.384152] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7740.727733] Lustre: server umount lustre-MDT0000 complete [ 7743.969909] LustreError: 234629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7743.993534] LustreError: 234629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7744.452616] LustreError: 234802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788593527 with bad export cookie 901305780870484817 [ 7744.457746] LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7744.463638] LustreError: 234802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7744.850540] Lustre: server umount lustre-MDT0001 complete [ 7757.484921] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 7760.770810] Key type lgssc unregistered [ 7761.044748] LNet: 235858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7761.052864] LNetError: 235858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7762.086770] LNet: Removed LNI 192.168.202.109@tcp [ 7762.951246] Key type .llcrypt unregistered [ 7762.959154] Key type ._llcrypt unregistered [ 7786.464406] Key type ._llcrypt registered [ 7786.469434] Key type .llcrypt registered [ 7786.576457] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 7788.014408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7788.041875] alg: No test for adler32 (adler32-zlib) [ 7789.198943] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 7789.476345] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 7791.209920] Key type lgssc registered [ 7792.431445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8064.791528] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 8067.298932] Key type lgssc unregistered [ 8067.772129] LNet: 237875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8067.780355] LNetError: 237875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8067.814837] LNet: Removed LNI 192.168.202.109@tcp [ 8068.871361] Key type .llcrypt unregistered [ 8068.872868] Key type ._llcrypt unregistered [ 8094.876753] Key type ._llcrypt registered [ 8094.879504] Key type .llcrypt registered [ 8094.974283] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing set_hostid [ 8111.552610] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing load_modules_local [ 8112.799389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8112.869078] alg: No test for adler32 (adler32-zlib) [ 8113.924270] Lustre: Lustre: Build Version: 2.17.58_3_g357faf0 [ 8114.132835] LNet: Added LNI 192.168.202.109@tcp [8/256/0/180] [ 8115.788641] Key type lgssc registered [ 8116.854263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8149.063360] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7870 sec ========== 03:38:50 (1788593930) [ 8151.411502] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:38:52 (1788593932) === [ 8156.040208] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:38:57 (1788593937) === [ 8201.810218] Lustre: DEBUG MARKER: oleg209-server.virtnet: executing unload_modules_local [ 8205.256685] Key type lgssc unregistered [ 8205.525625] LNet: 243110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8205.535556] LNetError: 243110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8205.572414] LNet: Removed LNI 192.168.202.109@tcp [ 8206.521450] Key type .llcrypt unregistered [ 8206.523355] Key type ._llcrypt unregistered