[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 557724168 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002502] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.006889] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009141] LSM: Security Framework initializing [ 0.010064] Yama: becoming mindful. [ 0.011051] SELinux: Initializing. [ 0.012087] *** VALIDATE selinux *** [ 0.020787] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025449] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026167] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027127] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029072] *** VALIDATE tmpfs *** [ 0.030497] *** VALIDATE proc *** [ 0.031274] *** VALIDATE cgroup *** [ 0.032012] *** VALIDATE cgroup2 *** [ 0.033300] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036032] Spectre V2 : User space: Vulnerable [ 0.037011] Speculative Store Bypass: Vulnerable [ 0.040201] debug: unmapping init [mem 0xffffffff9d459000-0xffffffff9d460fff] [ 0.042197] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043655] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044029] ... version: 2 [ 0.045015] ... bit width: 48 [ 0.046016] ... generic registers: 4 [ 0.047015] ... value mask: 0000ffffffffffff [ 0.048019] ... max period: 00007fffffffffff [ 0.049014] ... fixed-purpose events: 3 [ 0.050020] ... event mask: 000000070000000f [ 0.051310] rcu: Hierarchical SRCU implementation. [ 0.053548] smp: Bringing up secondary CPUs ... [ 0.054621] x86: Booting SMP configuration: [ 0.055031] .... node #0, CPUs: #1 #2 #3 [ 0.059121] smp: Brought up 1 node, 4 CPUs [ 0.061019] smpboot: Max logical packages: 1 [ 0.062017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.235733] node 0 deferred pages initialised in 171ms [ 0.242236] devtmpfs: initialized [ 0.243212] x86/mm: Memory block size: 128MB [ 0.245699] gcov: version magic: 0x41383552 [ 0.247494] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.250118] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.252400] pinctrl core: initialized pinctrl subsystem [ 0.253216] [ 0.253797] ************************************************************* [ 0.256019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.259018] ** ** [ 0.261018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.263018] ** ** [ 0.265018] ** This means that this kernel is built to expose internal ** [ 0.268019] ** IOMMU data structures, which may compromise security on ** [ 0.270016] ** your system. ** [ 0.273021] ** ** [ 0.275015] ** If you see this message and you are not debugging the ** [ 0.277019] ** kernel, report this immediately to your vendor! ** [ 0.281021] ** ** [ 0.283015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.286018] ************************************************************* [ 0.289891] NET: Registered protocol family 16 [ 0.291538] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.295093] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.297090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.301048] cpuidle: using governor menu [ 0.302832] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.303509] PCI: Using configuration type 1 for base access [ 0.307173] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.315142] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.319050] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.324067] cryptd: max_cpu_qlen set to 1000 [ 0.328322] ACPI: Added _OSI(Module Device) [ 0.329018] ACPI: Added _OSI(Processor Device) [ 0.331023] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.333018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.337387] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.345547] ACPI: Interpreter enabled [ 0.347076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.349019] ACPI: Using IOAPIC for interrupt routing [ 0.351146] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.354392] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.363509] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.365052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.368024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.371132] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.377673] acpiphp: Slot [2] registered [ 0.379151] acpiphp: Slot [5] registered [ 0.381173] acpiphp: Slot [6] registered [ 0.383135] acpiphp: Slot [7] registered [ 0.384141] acpiphp: Slot [8] registered [ 0.386122] acpiphp: Slot [9] registered [ 0.387180] acpiphp: Slot [10] registered [ 0.389135] acpiphp: Slot [3] registered [ 0.390291] acpiphp: Slot [4] registered [ 0.392129] acpiphp: Slot [11] registered [ 0.394245] acpiphp: Slot [12] registered [ 0.396258] acpiphp: Slot [13] registered [ 0.398158] acpiphp: Slot [14] registered [ 0.399160] acpiphp: Slot [15] registered [ 0.401161] acpiphp: Slot [16] registered [ 0.403144] acpiphp: Slot [17] registered [ 0.404163] acpiphp: Slot [18] registered [ 0.406221] acpiphp: Slot [19] registered [ 0.408184] acpiphp: Slot [20] registered [ 0.409134] acpiphp: Slot [21] registered [ 0.411120] acpiphp: Slot [22] registered [ 0.412135] acpiphp: Slot [23] registered [ 0.414125] acpiphp: Slot [24] registered [ 0.416163] acpiphp: Slot [25] registered [ 0.417115] acpiphp: Slot [26] registered [ 0.419109] acpiphp: Slot [27] registered [ 0.420104] acpiphp: Slot [28] registered [ 0.422137] acpiphp: Slot [29] registered [ 0.424104] acpiphp: Slot [30] registered [ 0.425168] acpiphp: Slot [31] registered [ 0.427072] PCI host bridge to bus 0000:00 [ 0.429029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.431048] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.434039] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.438039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.440027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.444037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.446251] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.448805] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.452397] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.464012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.469656] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.472024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.473029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.476026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.480649] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.484107] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.489060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.493811] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.499024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.511019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.518020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.523730] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.535020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.543022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.560024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.571989] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.581022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.592023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.614073] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.630316] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.638018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.646025] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.673028] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.686056] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.697023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.711023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.735020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.748000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.762022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.773023] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.826024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.843469] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.852020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.862021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.882025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.904712] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.906785] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.910754] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.913561] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.917273] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.922142] iommu: Default domain type: Passthrough [ 0.923774] SCSI subsystem initialized [ 0.925159] ACPI: bus type USB registered [ 0.927599] usbcore: registered new interface driver usbfs [ 0.930125] usbcore: registered new interface driver hub [ 0.932137] usbcore: registered new device driver usb [ 0.934269] pps_core: LinuxPPS API ver. 1 registered [ 0.936014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.940087] PTP clock support registered [ 0.942237] EDAC MC: Ver: 3.0.0 [ 0.946158] PCI: Using ACPI for IRQ routing [ 0.949460] NetLabel: Initializing [ 0.951015] NetLabel: domain hash size = 128 [ 0.953018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.955116] NetLabel: unlabeled traffic allowed by default [ 0.958175] vgaarb: loaded [ 0.960324] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.962018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.969000] clocksource: Switched to clocksource kvm-clock [ 1.094331] VFS: Disk quotas dquot_6.6.0 [ 1.096739] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.100465] *** VALIDATE ramfs *** [ 1.102209] *** VALIDATE hugetlbfs *** [ 1.105320] pnp: PnP ACPI init [ 1.109111] pnp: PnP ACPI: found 6 devices [ 1.127230] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.132841] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.135458] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.138553] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.141994] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.145125] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.149323] NET: Registered protocol family 2 [ 1.152206] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.157989] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.162798] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.168287] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.172287] TCP: Hash tables configured (established 65536 bind 65536) [ 1.175338] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.178829] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.181916] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.185140] NET: Registered protocol family 1 [ 1.190250] RPC: Registered named UNIX socket transport module. [ 1.192527] RPC: Registered udp transport module. [ 1.194431] RPC: Registered tcp transport module. [ 1.196479] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.198979] NET: Registered protocol family 44 [ 1.201542] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.204426] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.206846] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.209606] PCI: CLS 0 bytes, default 64 [ 1.212254] Unpacking initramfs... [ 2.730190] debug: unmapping init [mem 0xffff94893cc54000-0xffff94893ffbffff] [ 2.737195] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.739800] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.744130] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.250838] Initialise system trusted keyrings [ 3.252761] Key type blacklist registered [ 3.255348] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.266558] zbud: loaded [ 3.270313] *** VALIDATE nfs *** [ 3.271703] *** VALIDATE nfs4 *** [ 3.273599] pstore: using deflate compression [ 3.278498] Platform Keyring initialized [ 3.399329] NET: Registered protocol family 38 [ 3.401659] Key type asymmetric registered [ 3.403194] Asymmetric key parser 'x509' registered [ 3.404837] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.408227] io scheduler mq-deadline registered [ 3.410159] io scheduler kyber registered [ 3.411977] io scheduler bfq registered [ 3.414148] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.417284] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.420443] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.423757] ACPI: Power Button [PWRF] [ 3.429671] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.437153] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.454324] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.464775] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.493325] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.522859] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.552718] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.558255] Non-volatile memory driver v1.3 [ 3.560209] Linux agpgart interface v0.103 [ 3.592208] virtio_blk virtio1: [vda] 146168 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.595152] vda: detected capacity change from 0 to 74838016 [ 3.613703] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.619428] vdb: detected capacity change from 0 to 1073741824 [ 3.642489] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.648103] vdc: detected capacity change from 0 to 2621440000 [ 3.669828] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.674038] vdd: detected capacity change from 0 to 2621440000 [ 3.697484] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.700847] vde: detected capacity change from 0 to 4294967296 [ 3.723344] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.726100] vdf: detected capacity change from 0 to 4294967296 [ 3.738165] libphy: Fixed MDIO Bus: probed [ 3.743468] usbcore: registered new interface driver usbserial_generic [ 3.745898] usbserial: USB Serial support registered for generic [ 3.749334] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.756508] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.760131] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.762791] mousedev: PS/2 mouse device common for all mice [ 3.766819] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.768304] rtc_cmos 00:05: RTC can wake from S4 [ 3.777058] rtc_cmos 00:05: registered as rtc0 [ 3.778376] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.780757] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.780798] intel_pstate: CPU model not supported [ 3.787413] hid: raw HID events driver (C) Jiri Kosina [ 3.791179] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.792877] usbcore: registered new interface driver usbhid [ 3.792882] usbhid: USB HID core driver [ 3.793026] drop_monitor: Initializing network drop monitor service [ 3.794124] Initializing XFRM netlink socket [ 3.805753] NET: Registered protocol family 10 [ 3.808392] Segment Routing with IPv6 [ 3.809652] NET: Registered protocol family 17 [ 3.811508] mpls_gso: MPLS GSO support [ 3.818566] RAS: Correctable Errors collector initialized. [ 3.821603] AVX version of gcm_enc/dec engaged. [ 3.824302] AES CTR mode by8 optimization enabled [ 3.909398] sched_clock: Marking stable (3909342180, 0)->(4884980505, -975638325) [ 3.914340] registered taskstats version 1 [ 3.916309] Loading compiled-in X.509 certificates [ 3.917922] zswap: loaded using pool lzo/zbud [ 3.943067] Key type big_key registered [ 3.955786] Key type encrypted registered [ 3.956961] ima: No TPM chip found, activating TPM-bypass! [ 3.959276] ima: Allocated hash algorithm: sha1 [ 3.960802] ima: No architecture policies found [ 3.962421] evm: Initialising EVM extended attributes: [ 3.964321] evm: security.selinux [ 3.965517] evm: security.ima [ 3.966577] evm: security.capability [ 3.967882] evm: HMAC attrs: 0x1 [ 3.970596] rtc_cmos 00:05: setting system clock to 2026-08-24 08:31:21 UTC (1787560281) [ 3.977179] debug: unmapping init [mem 0xffffffff9e403000-0xffffffff9e5fffff] [ 3.980306] debug: unmapping init [mem 0xffffffff9d182000-0xffffffff9d458fff] [ 3.989093] Write protecting the kernel read-only data: 28672k [ 3.992896] debug: unmapping init [mem 0xffffffff9b803000-0xffffffff9b9fffff] [ 3.995836] debug: unmapping init [mem 0xffffffff9c114000-0xffffffff9c1fffff] [ 4.034748] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.043396] systemd[1]: Detected virtualization kvm. [ 4.044916] systemd[1]: Detected architecture x86-64. [ 4.046336] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.072404] systemd[1]: No hostname configured. [ 4.074157] systemd[1]: Set hostname to . [ 4.075839] random: systemd: uninitialized urandom read (16 bytes read) [ 4.077703] systemd[1]: Initializing machine ID from random generator. [ 4.149819] random: ln: uninitialized urandom read (6 bytes read) [ 4.257302] random: systemd: uninitialized urandom read (16 bytes read) [ 4.260619] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.268736] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 4.275282] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.027784] device-mapper: uevent: version 1.0.3 [ 5.030233] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 5.712759] random: fast init done [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.918065] virtio_net virtio0 ens2: renamed from eth0 [ 5.994941] scsi host0: ata_piix [ 6.046317] scsi host1: ata_piix [ 6.048349] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.051513] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.477083] random: crng init done [ 10.478777] random: 7 urandom warning(s) missed due to ratelimiting [ 10.490453] dracut-initqueue[598]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.164811] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.552386] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.861562] SELinux: Disabled at runtime. [ 12.926509] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.935502] systemd[1]: Detected virtualization kvm. [ 12.937398] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.483836] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.487495] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.493357] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.499454] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.504285] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.513178] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.517819] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. [ 13.705842] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.990995] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.346932] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.365181] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.581079] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.592759] EDAC sbridge: Ver: 1.1.2 [ 16.490548] Key type dns_resolver registered [ 16.801949] NFS: Registering the id_resolver key type [ 16.805369] Key type id_resolver registered [ 16.807549] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg442-server login: [ 33.349235] hrtimer: interrupt took 4993393 ns [ 74.181786] libcfs: loading out-of-tree module taints kernel. [ 74.278732] Key type ._llcrypt registered [ 74.294833] Key type .llcrypt registered [ 74.512992] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 96.392514] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 98.047937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 98.060704] alg: No test for adler32 (adler32-zlib) [ 100.010262] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 100.927057] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 102.607167] Key type lgssc registered [ 105.130687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 122.787067] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 174.172498] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 191.552592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 191.640650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.133521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.183632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.357201] Lustre: lustre-MDT0000: new disk, initializing [ 193.531354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.561697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 199.552290] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.580838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.831105] Lustre: 6521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 216.943846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 216.959281] Lustre: Skipped 1 previous similar message [ 217.123962] Lustre: lustre-MDT0001: new disk, initializing [ 217.209809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 217.251424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 217.268785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 222.137473] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.530380] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 239.946036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.264254] Lustre: lustre-OST0000: new disk, initializing [ 240.280586] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 240.289874] Lustre: 8460:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.367186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 245.831918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 245.843929] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 245.921778] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 247.513166] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.927499] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 263.079677] Lustre: lustre-OST0001: new disk, initializing [ 263.083237] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 263.090298] Lustre: 9532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 263.140821] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 269.306589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 269.316723] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 269.340713] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 269.428255] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 281.306375] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 289.510730] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 296.625422] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing check_logdir /tmp/testlogs/ [ 302.562040] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing yml_node [ 308.000140] Lustre: DEBUG MARKER: Client: 2.17.57.82 [ 310.679301] Lustre: DEBUG MARKER: MDS: 2.17.57.82 [ 313.589770] Lustre: DEBUG MARKER: OSS: 2.17.57.82 [ 315.466573] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 04:36:31 EDT 2026 [ 335.777348] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 337.609707] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 355.295950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 355.301310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.316946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 356.321461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 356.323097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 356.340792] Lustre: Skipped 1 previous similar message [ 356.368986] Lustre: Skipped 1 previous similar message [ 360.573597] Lustre: server umount lustre-MDT0000 complete [ 365.034551] LustreError: 8459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560642 with bad export cookie 3634962314116871943 [ 365.043661] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 365.045263] LustreError: 8459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 366.564126] LustreError: 6527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 366.564832] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.566589] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 366.566597] Lustre: Skipped 2 previous similar messages [ 366.582126] LustreError: 6527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 366.629369] Lustre: Skipped 2 previous similar messages [ 370.593121] LustreError: 6526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 370.595860] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 370.619170] LustreError: 6526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 370.634087] Lustre: Skipped 2 previous similar messages [ 374.693066] LustreError: 12172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.693167] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 374.720305] LustreError: 12172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 374.728143] Lustre: Skipped 1 previous similar message [ 379.359150] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 379.569198] Lustre: server umount lustre-MDT0001 complete [ 384.274499] Lustre: server umount lustre-OST0000 complete [ 388.941932] Lustre: server umount lustre-OST0001 complete [ 397.300158] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 407.357876] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 454.531760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 454.791605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 454.840644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 454.896226] Lustre: lustre-MDT0000: new disk, initializing [ 454.958313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 454.974317] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 460.565339] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 474.936911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.111274] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 475.207447] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 475.212405] Lustre: Skipped 1 previous similar message [ 475.370941] Lustre: lustre-MDT0001: new disk, initializing [ 475.575725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 475.628206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 475.650802] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 482.401127] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 494.111722] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 500.534288] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 508.705388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 509.256892] Lustre: lustre-OST0000: new disk, initializing [ 509.260534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 509.281867] Lustre: 17057:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 509.445497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 510.917285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 510.927341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 511.141116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 517.508091] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 528.848984] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 533.997748] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 534.253359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 538.896367] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 539.164593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 541.666893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.673175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 541.687665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 547.412507] Lustre: server umount lustre-OST0000 complete [ 551.906065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 551.926803] Lustre: Skipped 2 previous similar messages [ 557.838877] Lustre: server umount lustre-MDT0000 complete [ 562.147154] LustreError: 15961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 562.183364] LustreError: 15961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 563.296659] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560840 with bad export cookie 3634962314116873469 [ 563.338717] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 563.334248] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 563.621031] Lustre: server umount lustre-MDT0001 complete [ 568.725293] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 04:40:44 (1787560844) [ 578.163989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.826638] LustreError: 19321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.934703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 584.050993] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 589.283489] LustreError: 19322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 589.319607] LustreError: 19322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 594.797749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.027576] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 610.718988] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 616.659867] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 624.286248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 632.869944] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 642.390461] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 665.061898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 665.073329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 665.079250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 665.079260] Lustre: Skipped 7 previous similar messages [ 667.593966] Lustre: server umount lustre-OST0000 complete [ 679.137350] Lustre: server umount lustre-MDT0000 complete [ 680.416224] LustreError: 19317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 680.452105] LustreError: 19317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 684.279450] LustreError: 19303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787560961 with bad export cookie 3634962314116874358 [ 684.280836] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 684.299696] LustreError: 19303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 684.676688] Lustre: server umount lustre-MDT0001 complete [ 697.691166] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 701.338769] Key type lgssc unregistered [ 701.778742] LNet: 22880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.794984] LNetError: 22880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.815996] LNet: Removed LNI 192.168.204.142@tcp [ 702.799178] Key type .llcrypt unregistered [ 702.802084] Key type ._llcrypt unregistered [ 731.422559] Key type ._llcrypt registered [ 731.424361] Key type .llcrypt registered [ 731.640943] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 733.985274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 734.082544] alg: No test for adler32 (adler32-zlib) [ 735.416236] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 735.801567] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 737.519264] Key type lgssc registered [ 739.450248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 749.388702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 749.404556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 751.162264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 756.411104] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 761.328180] LustreError: 24295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 761.359373] LustreError: 24295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 766.438557] LustreError: 24294:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 766.907398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 767.477752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 773.672541] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 782.677731] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 789.604670] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 798.113057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 798.583214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 806.180544] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 815.266323] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 821.182598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 834.038602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 834.052440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 834.074374] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 835.564969] Lustre: server umount lustre-OST0000 complete [ 841.700630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 841.709381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 841.722098] Lustre: Skipped 1 previous similar message [ 841.738688] Lustre: Skipped 1 previous similar message [ 846.767682] Lustre: server umount lustre-MDT0000 complete [ 846.825741] LustreError: 24291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 846.871171] LustreError: 24291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 850.873291] LustreError: 24276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561128 with bad export cookie 15627280206864176516 [ 850.890997] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 861.404683] Lustre: server umount lustre-MDT0001 complete [ 875.733484] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 878.867397] Key type lgssc unregistered [ 879.140664] LNet: 28276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.151456] LNetError: 28276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.178491] LNet: Removed LNI 192.168.204.142@tcp [ 880.147200] Key type .llcrypt unregistered [ 880.149627] Key type ._llcrypt unregistered [ 907.733435] Key type ._llcrypt registered [ 907.736443] Key type .llcrypt registered [ 907.903478] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 909.568814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 909.821955] alg: No test for adler32 (adler32-zlib) [ 911.080362] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 911.536983] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 913.343198] Key type lgssc registered [ 914.890870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 924.135231] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 924.156113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 925.605523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 930.407558] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 935.909675] LustreError: 29697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 935.936774] LustreError: 29697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 939.767313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 940.138281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 945.108645] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 953.021452] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 959.038475] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 973.721697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 974.248534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 982.277862] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 991.402206] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 995.770514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1005.026837] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1005.039119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.064100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1006.051392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1006.055374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1006.079538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1010.816585] Lustre: server umount lustre-OST0000 complete [ 1016.288996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1016.294809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1016.336067] Lustre: Skipped 1 previous similar message [ 1016.354749] Lustre: Skipped 2 previous similar messages [ 1021.411855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1021.422671] Lustre: Skipped 1 previous similar message [ 1022.137669] Lustre: server umount lustre-MDT0000 complete [ 1026.532156] LustreError: 30701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1026.561463] LustreError: 30701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1026.599118] LustreError: 30431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561304 with bad export cookie 6589227958943676734 [ 1026.611588] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1026.626594] LustreError: 30431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1027.458486] Lustre: server umount lustre-MDT0001 complete [ 1047.349602] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1053.002815] Key type lgssc unregistered [ 1053.528959] LNet: 33619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.546960] LNetError: 33619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1053.602420] LNet: Removed LNI 192.168.204.142@tcp [ 1055.504108] Key type .llcrypt unregistered [ 1055.506840] Key type ._llcrypt unregistered [ 1083.770973] Key type ._llcrypt registered [ 1083.774636] Key type .llcrypt registered [ 1083.930910] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1085.237601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1085.254414] alg: No test for adler32 (adler32-zlib) [ 1086.403864] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 1086.696206] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1088.367472] Key type lgssc registered [ 1090.055411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1099.845904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1099.864970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1101.316914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1105.956891] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1110.518756] LustreError: 35034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1110.542094] LustreError: 35034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1115.617216] LustreError: 35033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1115.862261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1116.276086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1122.247545] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1130.576859] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1136.755656] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1144.213473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1144.734057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1145.786559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1151.348309] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1158.918339] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1171.427953] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1171.457465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1171.478557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1171.496985] Lustre: Skipped 1 previous similar message [ 1174.960272] Lustre: server umount lustre-OST0000 complete [ 1181.675187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.709117] Lustre: Skipped 1 previous similar message [ 1181.725574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1181.729105] Lustre: Skipped 1 previous similar message [ 1185.883411] Lustre: server umount lustre-MDT0000 complete [ 1186.784607] LustreError: 36875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1186.804173] LustreError: 36875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1190.064384] LustreError: 35015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561467 with bad export cookie 11009445810119082483 [ 1190.116770] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1190.492100] Lustre: server umount lustre-MDT0001 complete [ 1203.245569] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1206.331768] Key type lgssc unregistered [ 1206.790244] LNet: 38575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1206.804145] LNetError: 38575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1206.825788] LNet: Removed LNI 192.168.204.142@tcp [ 1208.166906] Key type .llcrypt unregistered [ 1208.169180] Key type ._llcrypt unregistered [ 1236.345886] Key type ._llcrypt registered [ 1236.347733] Key type .llcrypt registered [ 1236.499210] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1237.947990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1238.095858] alg: No test for adler32 (adler32-zlib) [ 1239.282584] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 1239.542266] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1241.217504] Key type lgssc registered [ 1242.372873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1249.967165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1249.995328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1251.654704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1256.049684] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1262.052210] LustreError: 39988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.079267] LustreError: 39988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1264.777851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1265.089540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1269.461746] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1276.895484] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1282.795461] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1289.938626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1290.395342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1296.580307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1297.494303] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1306.584608] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1317.344422] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1317.368797] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.402990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1319.513525] Lustre: lustre-OST0000: Not available for connect from 192.168.204.42@tcp (stopping) [ 1320.441022] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1320.451582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1324.574482] Lustre: lustre-OST0000: Not available for connect from 192.168.204.42@tcp (stopping) [ 1324.586238] Lustre: Skipped 2 previous similar messages [ 1329.691462] Lustre: lustre-OST0000: Not available for connect from 192.168.204.42@tcp (stopping) [ 1329.702532] Lustre: Skipped 2 previous similar messages [ 1331.167781] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1331.351206] Lustre: server umount lustre-OST0000 complete [ 1356.257954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1356.303531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1356.312658] Lustre: Skipped 3 previous similar messages [ 1359.945949] Lustre: server umount lustre-MDT0000 complete [ 1361.376750] LustreError: 39984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1361.422231] LustreError: 39984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1362.403142] LustreError: 39985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.415484] LustreError: 39985:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1364.394816] LustreError: 39969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561641 with bad export cookie 1536845986518908806 [ 1364.397152] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1364.407019] LustreError: 39969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1370.858091] Lustre: server umount lustre-MDT0001 complete [ 1385.346755] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1388.355589] Key type lgssc unregistered [ 1388.643906] LNet: 43704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.664298] LNetError: 43704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.701194] LNet: Removed LNI 192.168.204.142@tcp [ 1389.746366] Key type .llcrypt unregistered [ 1389.750378] Key type ._llcrypt unregistered [ 1414.798870] Key type ._llcrypt registered [ 1414.800690] Key type .llcrypt registered [ 1414.888651] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1416.411932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1416.439762] alg: No test for adler32 (adler32-zlib) [ 1417.556470] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 1417.773694] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1419.447218] Key type lgssc registered [ 1420.715172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1428.863765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1428.893847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1430.457441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1434.808376] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1439.725764] LustreError: 45119:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1439.755229] LustreError: 45119:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1443.488305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1443.851148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1448.644890] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1456.407763] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1463.858380] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1471.903839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1472.278264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1475.401017] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1479.811041] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1488.456112] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1496.641462] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.42@tcp (stopping) [ 1499.111668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1499.123167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1499.135468] Lustre: Skipped 1 previous similar message [ 1499.156884] Lustre: Skipped 2 previous similar messages [ 1501.731127] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.42@tcp (stopping) [ 1502.670842] Lustre: server umount lustre-MDT0000 complete [ 1504.230119] LustreError: 45846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1504.254838] LustreError: 45846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1506.610136] LustreError: 45099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787561784 with bad export cookie 12149806179125794012 [ 1506.629320] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1506.846051] LustreError: 45115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1506.886708] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.42@tcp (stopping) [ 1509.347714] LustreError: 46359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1509.353470] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1509.377020] Lustre: Skipped 1 previous similar message [ 1511.417701] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1511.427600] Lustre: Skipped 1 previous similar message [ 1513.101906] Lustre: server umount lustre-MDT0001 complete [ 1534.417213] Lustre: server umount lustre-OST0000 complete [ 1553.142662] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1556.632560] Key type lgssc unregistered [ 1556.930982] LNet: 48983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.948554] LNetError: 48983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1556.978553] LNet: Removed LNI 192.168.204.142@tcp [ 1558.107089] Key type .llcrypt unregistered [ 1558.109043] Key type ._llcrypt unregistered [ 1584.949899] Key type ._llcrypt registered [ 1584.951803] Key type .llcrypt registered [ 1585.090552] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1586.302496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1586.463405] alg: No test for adler32 (adler32-zlib) [ 1587.630769] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 1587.882226] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1589.561492] Key type lgssc registered [ 1591.121186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1600.804614] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1600.842507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1616.543510] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1621.663358] LustreError: 50370:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.142@tcp: failed processing log, type 4: rc = -110 [ 1648.351284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1654.749413] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1662.883493] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1828.805778] Lustre: server umount lustre-OST0000 complete [ 1844.660969] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1847.521740] Key type lgssc unregistered [ 1847.811815] LNet: 52063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1847.824692] LNetError: 52063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1847.838726] LNet: Removed LNI 192.168.204.142@tcp [ 1848.811214] Key type .llcrypt unregistered [ 1848.813730] Key type ._llcrypt unregistered [ 1871.939281] Key type ._llcrypt registered [ 1871.942789] Key type .llcrypt registered [ 1872.072030] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 1873.174749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1873.194466] alg: No test for adler32 (adler32-zlib) [ 1874.233678] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 1874.483813] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 1876.200216] Key type lgssc registered [ 1877.373965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1886.092992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1886.123079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1887.477451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.622846] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1896.933404] LustreError: 53478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1896.947129] LustreError: 53478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1900.678465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1901.018811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1905.940668] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1916.047108] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1924.716994] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1933.528820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1933.934899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1940.988147] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 1949.775363] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1956.843473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1956.844861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1956.874074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1957.922116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1958.508685] Lustre: server umount lustre-OST0000 complete [ 1962.985574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1963.007117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1963.015558] Lustre: Skipped 1 previous similar message [ 1967.586916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1967.601389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.618693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1969.071894] Lustre: server umount lustre-MDT0000 complete [ 1973.216437] LustreError: 53479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1973.236767] LustreError: 53479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1973.646439] LustreError: 53458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562251 with bad export cookie 9304012119834425642 [ 1973.650339] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1973.656905] LustreError: 53458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1974.065151] Lustre: server umount lustre-MDT0001 complete [ 1986.303898] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 1990.424366] Key type lgssc unregistered [ 1990.783816] LNet: 57012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1990.803681] LNetError: 57012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1990.858753] LNet: Removed LNI 192.168.204.142@tcp [ 1992.211178] Key type .llcrypt unregistered [ 1992.216369] Key type ._llcrypt unregistered [ 2018.733970] Key type ._llcrypt registered [ 2018.736105] Key type .llcrypt registered [ 2018.825615] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2019.986311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2020.012958] alg: No test for adler32 (adler32-zlib) [ 2021.236578] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 2021.679684] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2023.399596] Key type lgssc registered [ 2024.812481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2033.367569] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2033.392468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2048.929424] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2054.047325] LustreError: 58400:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.142@tcp: failed processing log, type 4: rc = -110 [ 2080.799256] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2086.709839] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2094.662385] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2101.572576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2101.694689] LustreError: 57736:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002e19b373 x1874394925238528/t0(0) o250->MGC192.168.204.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2102.167824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2102.247508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2106.226165] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2107.363581] LustreError: 58426:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.379142] LustreError: 58426:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2112.486460] LustreError: 58425:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.798819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.086363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2119.515436] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2128.379801] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2134.648846] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2138.085587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2138.106048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.135039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2140.645075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2140.657761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2140.676455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2141.679221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2141.696165] Lustre: Skipped 1 previous similar message [ 2146.787497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2146.801297] Lustre: Skipped 1 previous similar message [ 2151.391165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2151.656691] Lustre: server umount lustre-OST0000 complete [ 2176.481901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2176.489159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2176.508287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2181.177980] Lustre: server umount lustre-MDT0000 complete [ 2183.650496] LustreError: 59438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2183.668628] LustreError: 59438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2185.023620] LustreError: 58406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562462 with bad export cookie 4374307873972418117 [ 2185.034874] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2185.037893] LustreError: 58406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2191.424410] Lustre: server umount lustre-MDT0001 complete [ 2203.134597] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2205.915403] Key type lgssc unregistered [ 2206.175934] LNet: 62151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.190311] LNetError: 62151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2207.213201] LNet: Removed LNI 192.168.204.142@tcp [ 2207.988418] Key type .llcrypt unregistered [ 2207.991391] Key type ._llcrypt unregistered [ 2231.087993] Key type ._llcrypt registered [ 2231.092879] Key type .llcrypt registered [ 2231.191321] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2232.426410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2232.463640] alg: No test for adler32 (adler32-zlib) [ 2233.564551] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 2233.865612] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2235.575212] Key type lgssc registered [ 2236.818476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2245.142339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2245.156405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2246.535293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2251.260244] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2256.866737] LustreError: 63564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2256.894420] LustreError: 63564:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2261.491785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2262.128943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2267.318302] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2276.005848] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2283.987673] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2292.573300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2292.938804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2299.621316] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2308.224469] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2316.526493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2327.018887] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2327.024806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2327.030001] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2327.039709] Lustre: Skipped 1 previous similar message [ 2330.830849] Lustre: server umount lustre-OST0000 complete [ 2337.252327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2337.260201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2337.277970] Lustre: Skipped 1 previous similar message [ 2337.300586] Lustre: Skipped 2 previous similar messages [ 2340.930265] Lustre: server umount lustre-MDT0000 complete [ 2342.370328] LustreError: 63559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.393667] LustreError: 63559:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2344.595390] LustreError: 63544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562622 with bad export cookie 5933139713409362391 [ 2344.625554] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2345.229773] Lustre: server umount lustre-MDT0001 complete [ 2359.316422] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2364.189328] Key type lgssc unregistered [ 2364.770576] LNet: 67125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2364.793400] LNetError: 67125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2364.811071] LNet: Removed LNI 192.168.204.142@tcp [ 2366.390967] Key type .llcrypt unregistered [ 2366.399417] Key type ._llcrypt unregistered [ 2395.756506] Key type ._llcrypt registered [ 2395.760709] Key type .llcrypt registered [ 2395.870093] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2397.324301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2397.385602] alg: No test for adler32 (adler32-zlib) [ 2398.558383] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 2398.773313] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2400.463193] Key type lgssc registered [ 2401.821252] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2411.280058] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 05:11:27 (1787562687) [ 2418.618192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2418.657662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2420.056281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2424.984649] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2430.434547] LustreError: 68830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.458882] LustreError: 68830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2435.047623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2435.692349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2441.556797] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2451.127978] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2457.617085] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2464.806931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2465.188088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.227493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2471.755183] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2479.845549] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2487.709422] Lustre: Failing over lustre-MDT0000 [ 2487.857902] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.42@tcp (stopping) [ 2488.368286] Lustre: server umount lustre-MDT0000 complete [ 2490.872852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.888836] LustreError: 68827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2490.910908] Lustre: Skipped 2 previous similar messages [ 2490.928952] LustreError: 68827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2492.970266] LustreError: 70725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2495.978151] LustreError: 68831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.013531] LustreError: 68831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2499.969571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2500.055569] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2500.181127] Lustre: *** cfs_fail_loc=135, val=0*** [ 2500.267078] LustreError: 71920:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2500.276448] LustreError: 71920:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.142@tcp: cfg command failed: rc = -2 [ 2500.296827] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2500.296827] [ 2500.301581] LustreError: MGC192.168.204.142@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2500.310134] LustreError: 71908:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2500.315803] LustreError: 71908:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 2500.326429] LustreError: 71908:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2500.333660] LustreError: 71908:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2500.355958] Lustre: server umount lustre-MDT0000 complete [ 2500.358550] LustreError: 71908:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2501.101607] LustreError: 68830:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2501.127305] LustreError: 68830:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2509.039573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2509.187830] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2509.198360] LustreError: 68822:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2509.257188] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.42@tcp (not set up) [ 2509.551171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2509.633972] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2513.307595] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2514.510174] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2514.923645] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2514.979276] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2515.043254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2525.156636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2525.169861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2525.174899] Lustre: Skipped 1 previous similar message [ 2528.655889] Lustre: server umount lustre-OST0000 complete [ 2535.397220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2535.402891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2535.429468] Lustre: Skipped 1 previous similar message [ 2535.436261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2538.384042] Lustre: server umount lustre-MDT0000 complete [ 2540.519316] LustreError: 68826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.548765] LustreError: 68826:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2542.291469] LustreError: 72298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562819 with bad export cookie 5211602594539013104 [ 2542.295759] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2542.302190] LustreError: 72298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2542.582826] Lustre: server umount lustre-MDT0001 complete [ 2554.892175] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2558.126719] Key type lgssc unregistered [ 2558.454351] LNet: 73705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2558.466431] LNetError: 73705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2558.488199] LNet: Removed LNI 192.168.204.142@tcp [ 2559.429848] Key type .llcrypt unregistered [ 2559.433454] Key type ._llcrypt unregistered [ 2582.625365] Key type ._llcrypt registered [ 2582.631906] Key type .llcrypt registered [ 2582.742765] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2583.723109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2583.872757] alg: No test for adler32 (adler32-zlib) [ 2584.944196] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 2585.289640] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2587.007156] Key type lgssc registered [ 2588.241529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2595.869240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2595.887756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.380766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2601.606681] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2606.570381] LustreError: 75122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.604992] LustreError: 75122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2611.077187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.746937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2616.289635] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2624.028633] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2630.441728] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2636.921475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2637.242763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2642.435484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2642.914632] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2650.627992] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2656.132259] Lustre: Failing over lustre-MDT0000 [ 2656.472281] Lustre: server umount lustre-MDT0000 complete [ 2657.326299] LustreError: 76130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2657.761946] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2657.767612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2657.772164] Lustre: Skipped 1 previous similar message [ 2662.432239] LustreError: 75117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.455404] LustreError: 75117:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2665.927190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2666.085688] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2666.306160] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2666.532226] LustreError: 78211:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2666.540358] LustreError: 78211:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.142@tcp: cfg command failed: rc = -9 [ 2666.553823] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2666.553823] [ 2666.570690] LustreError: MGC192.168.204.142@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2666.584188] LustreError: 78199:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2666.590865] LustreError: 78199:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -9 [ 2666.607433] LustreError: 78199:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2666.615769] LustreError: 78199:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2666.639880] Lustre: server umount lustre-MDT0000 complete [ 2666.644843] LustreError: 78199:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2667.548194] LustreError: 78032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2667.567349] LustreError: 78032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2672.668583] LustreError: 75118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.690769] LustreError: 75118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2674.915421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2675.026643] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2675.274967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2675.315203] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2678.812905] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2678.849280] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2680.326829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2680.344611] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2680.385479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2689.505941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2689.510367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2689.531446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2690.533047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.537877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2690.556832] Lustre: Skipped 1 previous similar message [ 2692.387289] Lustre: server umount lustre-OST0000 complete [ 2700.771130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2700.775378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2700.779165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2702.440905] Lustre: server umount lustre-MDT0000 complete [ 2705.662511] LustreError: 75856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787562983 with bad export cookie 1843747659116771510 [ 2705.667052] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2705.672638] LustreError: 75856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2706.047214] Lustre: server umount lustre-MDT0001 complete [ 2717.959407] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2721.845983] Key type lgssc unregistered [ 2722.299375] LNet: 79994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2722.320545] LNetError: 79994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2722.338385] LNet: Removed LNI 192.168.204.142@tcp [ 2723.723674] Key type .llcrypt unregistered [ 2723.725655] Key type ._llcrypt unregistered [ 2739.715740] Key type ._llcrypt registered [ 2739.720297] Key type .llcrypt registered [ 2739.834416] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2740.062507] Key type .llcrypt unregistered [ 2740.068709] Key type ._llcrypt unregistered [ 2763.489636] Key type ._llcrypt registered [ 2763.492128] Key type .llcrypt registered [ 2763.566722] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2764.775367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2764.805392] alg: No test for adler32 (adler32-zlib) [ 2765.939524] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 2766.253724] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2768.047173] Key type lgssc registered [ 2769.230954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2776.589649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2776.616397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2778.009903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2782.145739] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2788.324073] LustreError: 81914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2788.341850] LustreError: 81914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2789.830803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2790.136223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2794.305738] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2800.624657] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2805.928508] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2811.107351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2811.416693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2816.480397] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2822.722570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2824.140360] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2834.913519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2834.923166] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2834.935912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2837.989431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2837.996037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2838.009838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2840.142131] Lustre: server umount lustre-OST0000 complete [ 2846.696795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2846.705804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.730407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2846.735557] Lustre: Skipped 1 previous similar message [ 2850.009787] Lustre: server umount lustre-MDT0000 complete [ 2853.346332] LustreError: 81914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.390908] LustreError: 81914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2853.727417] LustreError: 81896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563131 with bad export cookie 10458173193663245367 [ 2853.747848] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2854.258274] Lustre: server umount lustre-MDT0001 complete [ 2868.351684] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 2871.402351] Key type lgssc unregistered [ 2871.725835] LNet: 85450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2871.735306] LNetError: 85450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2871.745709] LNet: Removed LNI 192.168.204.142@tcp [ 2872.564100] Key type .llcrypt unregistered [ 2872.569569] Key type ._llcrypt unregistered [ 2893.200761] Key type ._llcrypt registered [ 2893.203479] Key type .llcrypt registered [ 2893.283985] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 2894.060446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2894.097913] alg: No test for adler32 (adler32-zlib) [ 2895.112050] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 2895.317757] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 2896.999833] Key type lgssc registered [ 2898.102664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2906.463185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2906.478650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2908.056519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2913.123609] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2918.370162] LustreError: 86842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.381645] LustreError: 86842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2923.482235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2923.488752] LustreError: 86843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2924.084413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2929.830852] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2936.605733] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2944.233381] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2952.071133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2952.629859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2957.847900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2960.380395] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 2967.944995] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2974.177337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2974.183066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2974.191826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2974.194677] Lustre: Skipped 1 previous similar message [ 2974.205403] Lustre: Skipped 1 previous similar message [ 2975.977877] Lustre: server umount lustre-OST0000 complete [ 2980.322104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2980.329546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2980.350144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2984.418105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2984.419973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2986.588851] Lustre: server umount lustre-MDT0000 complete [ 2989.546521] LustreError: 86839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2989.580755] LustreError: 86839:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2990.588659] LustreError: 86824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563268 with bad export cookie 4596276943074501325 [ 2990.593122] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2990.603833] LustreError: 86824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2990.918952] Lustre: server umount lustre-MDT0001 complete [ 3000.910928] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 3004.004995] Key type lgssc unregistered [ 3004.381928] LNet: 90380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3004.395642] LNetError: 90380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3004.411616] LNet: Removed LNI 192.168.204.142@tcp [ 3005.422942] Key type .llcrypt unregistered [ 3005.426442] Key type ._llcrypt unregistered [ 3029.888520] Key type ._llcrypt registered [ 3029.895454] Key type .llcrypt registered [ 3030.038694] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3031.256313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3031.344201] alg: No test for adler32 (adler32-zlib) [ 3032.392981] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 3032.600569] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 3034.263228] Key type lgssc registered [ 3035.411656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3045.128836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3045.151610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3046.970400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3052.485514] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3057.133243] LustreError: 91794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3057.157733] LustreError: 91794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3062.278586] LustreError: 91795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3062.358514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3062.784959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3067.016059] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3074.484440] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3080.029739] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3087.193868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3087.522040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3091.665266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 3093.265673] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3100.608580] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3117.542355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3117.545522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.551816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3117.596049] Lustre: Skipped 1 previous similar message [ 3122.657763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3122.663231] Lustre: Skipped 1 previous similar message [ 3123.096394] Lustre: server umount lustre-OST0000 complete [ 3127.779930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3127.788290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3127.814051] Lustre: Skipped 1 previous similar message [ 3127.822129] Lustre: Skipped 1 previous similar message [ 3132.897470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3132.903371] Lustre: Skipped 2 previous similar messages [ 3133.664076] Lustre: server umount lustre-MDT0000 complete [ 3137.806653] LustreError: 91773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563415 with bad export cookie 13697344544772278206 [ 3137.807808] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3137.821977] LustreError: 91773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3138.200385] Lustre: server umount lustre-MDT0001 complete [ 3148.111728] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 3150.803624] Key type lgssc unregistered [ 3151.138815] LNet: 95352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3151.143140] LNetError: 95352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3152.175264] LNet: Removed LNI 192.168.204.142@tcp [ 3153.105046] Key type .llcrypt unregistered [ 3153.106762] Key type ._llcrypt unregistered [ 3175.423589] Key type ._llcrypt registered [ 3175.426926] Key type .llcrypt registered [ 3175.517627] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3176.651438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3176.673218] alg: No test for adler32 (adler32-zlib) [ 3177.807557] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 3178.040728] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 3179.711185] Key type lgssc registered [ 3180.944983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3188.908411] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3188.936331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3204.575662] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3209.695409] LustreError: 96738:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.142@tcp: failed processing log, type 4: rc = -110 [ 3236.385514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3242.787869] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3250.371584] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3255.361265] Lustre: server umount lustre-OST0000 complete [ 3263.116710] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 05:25:39 (1787563539) [ 3268.923831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.337865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3273.191893] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3274.725643] LustreError: 98379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3274.747687] LustreError: 98379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3278.830548] LustreError: 98378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3281.184191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3281.501874] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3284.930561] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3291.939869] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3298.101658] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3304.417212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3304.741112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3311.388737] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3314.060431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3319.566026] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3334.644417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3334.648865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3334.669495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3334.680859] Lustre: Skipped 1 previous similar message [ 3334.691677] Lustre: Skipped 1 previous similar message [ 3337.958586] Lustre: server umount lustre-OST0000 complete [ 3342.819468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3342.834281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3342.865165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3344.868725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3344.869712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3348.448952] Lustre: server umount lustre-MDT0000 complete [ 3349.991840] LustreError: 98379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3350.020787] LustreError: 98379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3351.855079] LustreError: 98359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563629 with bad export cookie 15745038060584269718 [ 3351.858063] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3351.862429] LustreError: 98359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3352.279957] Lustre: server umount lustre-MDT0001 complete [ 3363.594793] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 3366.848029] Key type lgssc unregistered [ 3367.158959] LNet: 102073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3367.164243] LNetError: 102073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3367.181798] LNet: Removed LNI 192.168.204.142@tcp [ 3368.032580] Key type .llcrypt unregistered [ 3368.036191] Key type ._llcrypt unregistered [ 3413.871319] Key type ._llcrypt registered [ 3413.875828] Key type .llcrypt registered [ 3414.017521] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3415.249484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3415.321676] alg: No test for adler32 (adler32-zlib) [ 3416.484473] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 3416.701450] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 3418.447185] Key type lgssc registered [ 3420.281090] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3428.423584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3428.462256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3429.899415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3434.902529] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3440.096723] LustreError: 104261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3440.122173] LustreError: 104261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3443.340289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3443.758922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3448.652369] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3457.661495] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3464.907776] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3471.487499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3471.827055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3474.952326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3477.996612] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3485.978583] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3500.014108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3500.023701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3500.043357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3504.614655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3504.622514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3504.633644] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3505.990209] Lustre: server umount lustre-OST0000 complete [ 3510.240362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3510.253616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3510.281146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3510.297957] Lustre: Skipped 1 previous similar message [ 3514.850180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3514.855282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3516.372087] Lustre: server umount lustre-MDT0000 complete [ 3519.969917] LustreError: 104262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3519.991145] LustreError: 104262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3520.288160] LustreError: 104242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787563797 with bad export cookie 2942236825379745645 [ 3520.303671] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3520.316931] LustreError: 104242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3520.690372] Lustre: server umount lustre-MDT0001 complete [ 3533.165772] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 3536.057712] Key type lgssc unregistered [ 3536.361147] LNet: 107820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3536.371383] LNetError: 107820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3536.393275] LNet: Removed LNI 192.168.204.142@tcp [ 3537.219830] Key type .llcrypt unregistered [ 3537.224521] Key type ._llcrypt unregistered [ 3555.583788] Key type ._llcrypt registered [ 3555.586699] Key type .llcrypt registered [ 3555.724285] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3556.909782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3556.939605] alg: No test for adler32 (adler32-zlib) [ 3557.956614] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 3558.190891] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 3559.903325] Key type lgssc registered [ 3561.019880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3568.963684] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3569.004609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3584.607441] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3589.728776] LustreError: 109062:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.142@tcp: failed processing log, type 4: rc = -110 [ 3616.416137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3623.353653] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3633.075218] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3640.818642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3641.052139] LustreError: MGC192.168.204.142@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3641.060924] LustreError: 110086:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3641.067411] LustreError: 110086:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 3641.093263] LustreError: 110086:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3641.249876] Lustre: server umount lustre-MDT0000 complete [ 3641.256052] LustreError: 110086:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3659.679176] Lustre: 108400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787563921/real 1787563921] req@000000004c13e44e x1874396535852416/t0(0) o400->MGC192.168.204.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787563937 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3659.705812] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3659.719857] LustreError: 109074:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 3659.855822] Lustre: server umount lustre-OST0000 complete [ 3671.669408] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 3682.493366] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3724.577208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3724.978292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3725.020282] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3725.119602] Lustre: lustre-MDT0000: new disk, initializing [ 3725.236562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3725.272569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3730.612180] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3742.168084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3742.263853] Lustre: 112889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3742.311358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3742.315215] Lustre: Skipped 1 previous similar message [ 3742.371894] Lustre: lustre-MDT0001: new disk, initializing [ 3742.467390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3742.504079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3742.523074] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3748.457371] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3758.361401] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3764.742547] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3775.084724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3775.678816] Lustre: lustre-OST0000: new disk, initializing [ 3775.691935] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3775.710533] Lustre: 114856:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3775.880817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3782.617658] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3784.244344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3784.258581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3784.434650] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3793.236758] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3798.868179] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3799.074507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3804.277223] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3804.536391] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3810.279265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3810.280635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3810.309541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3813.113507] Lustre: server umount lustre-OST0000 complete [ 3819.493593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3819.498979] LustreError: Skipped 1 previous similar message [ 3819.503110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3819.514273] Lustre: Skipped 1 previous similar message [ 3819.523366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3819.537319] Lustre: Skipped 1 previous similar message [ 3820.521934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.524930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3820.558799] Lustre: Skipped 1 previous similar message [ 3823.625384] Lustre: server umount lustre-MDT0000 complete [ 3827.675144] LustreError: 113683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564105 with bad export cookie 11289699043133216391 [ 3827.683832] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3827.685784] LustreError: 113683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3828.106176] Lustre: server umount lustre-MDT0001 complete [ 3838.590480] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 05:35:14 (1787564114) [ 3842.733027] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3863.474244] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 3873.608751] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 3911.954983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.317829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3912.342444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3912.420961] Lustre: lustre-MDT0000: new disk, initializing [ 3912.522642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3912.545395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3918.318951] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3930.148525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.364425] Lustre: 119759:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3930.444195] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3930.450881] Lustre: Skipped 1 previous similar message [ 3930.548661] Lustre: lustre-MDT0001: new disk, initializing [ 3930.677309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3930.733542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3930.744821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3935.512289] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3946.168499] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3952.282775] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3959.695812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3959.898587] Lustre: lustre-OST0000: new disk, initializing [ 3959.901477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3959.906995] Lustre: 121728:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3959.981556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3965.920751] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 3966.555907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3966.573286] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3966.684328] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3977.149466] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3983.237148] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3983.572614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3989.476529] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3989.718893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3992.546931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3992.551094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3992.555264] LustreError: Skipped 1 previous similar message [ 3992.560079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3992.568090] Lustre: Skipped 1 previous similar message [ 3997.664996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3997.681829] Lustre: Skipped 1 previous similar message [ 4005.855248] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4005.982238] Lustre: server umount lustre-OST0000 complete [ 4012.512363] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4012.529304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4012.552671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4012.562063] Lustre: Skipped 3 previous similar messages [ 4015.888846] Lustre: server umount lustre-MDT0000 complete [ 4018.145691] LustreError: 119766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4018.160823] LustreError: 119766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4020.085938] LustreError: 119753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564297 with bad export cookie 11289699043133217273 [ 4020.096716] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4020.108828] LustreError: 119753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4020.615984] Lustre: server umount lustre-MDT0001 complete [ 4030.106340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4030.548464] LustreError: 123946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4030.621532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4035.953110] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4036.065404] LustreError: 123947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4041.187868] LustreError: 123946:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4046.314905] LustreError: 123947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4046.811178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4052.258843] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4060.768790] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4066.959781] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4074.461761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4082.619044] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4091.827814] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4100.011597] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4106.720638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4106.727816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4106.752116] Lustre: Skipped 1 previous similar message [ 4106.760821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4106.769746] Lustre: Skipped 2 previous similar messages [ 4111.242651] Lustre: server umount lustre-OST0000 complete [ 4121.756823] Lustre: server umount lustre-MDT0000 complete [ 4123.108052] LustreError: 123940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4123.123264] LustreError: 123940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4125.544844] LustreError: 123927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564403 with bad export cookie 11289699043133218155 [ 4125.553747] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4125.558622] LustreError: 123927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4125.989710] Lustre: server umount lustre-MDT0001 complete [ 4137.717617] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 4140.938680] Key type lgssc unregistered [ 4141.301436] LNet: 127580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4141.313983] LNetError: 127580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4141.335596] LNet: Removed LNI 192.168.204.142@tcp [ 4142.334822] Key type .llcrypt unregistered [ 4142.339222] Key type ._llcrypt unregistered [ 4164.166888] Key type ._llcrypt registered [ 4164.171089] Key type .llcrypt registered [ 4164.288728] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 4179.396025] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 4180.613945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4180.660150] alg: No test for adler32 (adler32-zlib) [ 4182.248992] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 4182.571635] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 4184.407218] Key type lgssc registered [ 4186.251304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4224.562737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4224.591057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4226.014844] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4226.068493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4226.159076] Lustre: lustre-MDT0000: new disk, initializing [ 4226.342033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4226.364780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4231.517355] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4244.384610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4244.591552] Lustre: 130735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4244.671410] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4244.676566] Lustre: Skipped 1 previous similar message [ 4244.750096] Lustre: lustre-MDT0001: new disk, initializing [ 4244.886535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4244.949216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4244.956100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4249.488674] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4259.260929] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4265.591255] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4273.054496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4273.486938] Lustre: lustre-OST0000: new disk, initializing [ 4273.495144] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4273.505602] Lustre: 132701:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4273.638295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4281.412209] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4281.930858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4281.957346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4282.156246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4293.021782] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4299.236772] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4299.650706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4307.357587] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4307.814231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4313.059361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4313.061352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4313.078163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4313.083967] LustreError: Skipped 1 previous similar message [ 4313.097883] Lustre: Skipped 1 previous similar message [ 4316.491896] Lustre: server umount lustre-OST0000 complete [ 4321.764493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4321.776710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4321.802169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4321.810469] Lustre: Skipped 1 previous similar message [ 4323.297635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4323.300977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4328.417076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4328.438692] Lustre: Skipped 2 previous similar messages [ 4332.514956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4332.528884] Lustre: Skipped 1 previous similar message [ 4335.071154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4335.287677] Lustre: server umount lustre-MDT0000 complete [ 4339.807654] LustreError: 130726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564617 with bad export cookie 16468171220946364431 [ 4339.815284] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4339.832724] LustreError: 130726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4340.350220] Lustre: server umount lustre-MDT0001 complete [ 4351.334490] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 05:43:46 (1787564626) [ 4360.398376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4360.870617] LustreError: 135118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4360.890520] LustreError: 135118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4360.980316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4365.588488] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4366.310536] LustreError: 135119:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4371.431904] LustreError: 135118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4375.565730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4375.890942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4380.919081] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4389.531526] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4395.985207] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4397.845059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.867112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4397.879536] Lustre: Skipped 1 previous similar message [ 4412.384164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4412.630691] Lustre: server umount lustre-MDT0000 complete [ 4416.482419] LustreError: 135112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4416.508911] LustreError: 135112:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4417.019334] LustreError: 135098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564694 with bad export cookie 16468171220946365313 [ 4417.020831] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4417.030978] LustreError: 135098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4417.561288] Lustre: server umount lustre-MDT0001 complete [ 4427.669785] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 05:45:03 (1787564703) [ 4435.210677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4450.847658] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4455.967768] LustreError: 137536:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.142@tcp: failed processing log, type 4: rc = -110 [ 4481.695309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4488.665369] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4496.804594] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4499.150523] Lustre: server umount lustre-OST0000 complete [ 4508.205844] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 05:46:24 (1787564784) [ 4515.193138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4515.495939] LustreError: 138981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4515.567056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4519.870189] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4526.051917] LustreError: 138982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4526.069776] LustreError: 138982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4529.124501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4533.690515] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4542.082692] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4548.391278] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4555.723507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4555.998636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4556.002743] Lustre: Skipped 1 previous similar message [ 4562.310121] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4571.137925] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4583.886736] Lustre: server umount lustre-MDT0000 complete [ 4585.958141] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4585.970415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4585.977691] Lustre: Skipped 1 previous similar message [ 4585.996594] LustreError: 138978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4586.022529] LustreError: 138978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4587.538261] LustreError: 138962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564865 with bad export cookie 16468171220946365971 [ 4587.549132] LustreError: 138962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4587.549368] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4587.927167] Lustre: server umount lustre-MDT0001 complete [ 4603.147850] Lustre: server umount lustre-OST0000 complete [ 4613.278709] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 05:48:09 (1787564889) [ 4621.221611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4621.609540] LustreError: 142658:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4621.752581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4625.773969] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4634.732707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4640.306573] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4649.085968] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4655.576940] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4662.805923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4667.194075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4669.471835] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4677.220610] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4681.824775] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4682.050295] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4686.184134] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4686.440884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4690.917159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4690.917185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4690.934866] Lustre: Skipped 2 previous similar messages [ 4690.956192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4690.961398] Lustre: Skipped 7 previous similar messages [ 4694.326366] Lustre: server umount lustre-OST0000 complete [ 4704.200340] Lustre: server umount lustre-MDT0000 complete [ 4706.278068] LustreError: 142655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4706.311274] LustreError: 142655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4707.589047] LustreError: 142638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787564985 with bad export cookie 16468171220946367042 [ 4707.596296] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4707.601021] LustreError: 142638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4707.859158] Lustre: server umount lustre-MDT0001 complete [ 4715.721369] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 05:49:52 (1787564992) [ 4721.692842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4737.247776] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4742.367511] LustreError: 146817:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.142@tcp: failed processing log, type 4: rc = -110 [ 4768.095441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4768.100610] Lustre: Skipped 2 previous similar messages [ 4774.240636] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4781.932098] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4788.118448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4788.920780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4793.164461] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4801.415545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4805.594382] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4812.467409] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4818.184019] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4822.808588] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4823.025097] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4827.792591] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4828.070216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4832.738101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4832.751913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4832.765897] Lustre: Skipped 2 previous similar messages [ 4832.774500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4832.777929] Lustre: Skipped 2 previous similar messages [ 4836.446641] Lustre: server umount lustre-MDT0000 complete [ 4838.886868] LustreError: 149156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4838.906321] LustreError: 149156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4840.955593] LustreError: 146825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787565118 with bad export cookie 16468171220946368071 [ 4840.963635] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4840.974724] LustreError: 146825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4841.344842] Lustre: server umount lustre-MDT0001 complete [ 4855.705763] Lustre: server umount lustre-OST0000 complete [ 4865.245969] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 05:52:21 (1787565141) [ 4872.817227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4888.481293] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4893.599514] LustreError: 151012:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.142@tcp: failed processing log, type 4: rc = -110 [ 4925.071268] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4932.018329] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4938.959297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4939.510397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 4943.749578] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4952.606471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4957.602704] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4966.531582] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4972.626284] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4979.598915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4979.794506] Lustre: lustre-OST0001: new disk, initializing [ 4979.800506] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4979.810330] Lustre: 153887:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4986.343567] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 4989.991912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4989.999936] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4990.111157] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4995.879828] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5000.005520] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5000.207021] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5004.538100] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5004.747529] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5009.385279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5009.405808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5009.427384] Lustre: Skipped 2 previous similar messages [ 5009.442940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5009.447167] Lustre: Skipped 3 previous similar messages [ 5012.756916] Lustre: server umount lustre-OST0000 complete [ 5030.367140] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5030.502675] Lustre: server umount lustre-OST0001 complete [ 5034.991604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5035.002488] LustreError: Skipped 2 previous similar messages [ 5040.597087] Lustre: server umount lustre-MDT0000 complete [ 5044.511474] LustreError: 151019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787565322 with bad export cookie 16468171220946369023 [ 5044.525225] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5044.527870] LustreError: 151019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5044.782611] Lustre: server umount lustre-MDT0001 complete [ 5068.232634] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 05:55:44 (1787565344) [ 5069.941454] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5071.659762] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5073.228650] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 05:55:49 (1787565349) [ 5080.255023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5080.571536] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5080.600330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5080.906692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5080.914796] Lustre: Skipped 6 previous similar messages [ 5086.417464] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5097.401909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5097.489868] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5097.508396] Lustre: 157446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5102.840924] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5112.648403] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5121.658625] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5130.909017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5131.165763] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5140.112426] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5141.166748] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5152.000478] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5158.272450] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5158.692328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5164.237945] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5164.722850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5171.680357] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5171.694684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5171.710170] Lustre: Skipped 5 previous similar messages [ 5171.728095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5171.737366] Lustre: Skipped 11 previous similar messages [ 5173.676473] Lustre: server umount lustre-OST0000 complete [ 5198.685376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5206.034589] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5211.633298] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5211.652784] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5214.237334] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5220.154969] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5220.386313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5221.929160] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5221.948137] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5224.875926] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5225.096936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5231.507514] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff98d54867e000.ost_server_uuid 50 [ 5233.381553] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff98d54867e000.ost_server_uuid in FULL state after 0 sec [ 5233.589544] LustreError: 160587:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.204.42@tcp) returned error from blocking AST (req@000000006769ca1e x1874397191480960 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000f2502be1/0xe48ab2792cd1c359 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.42@tcp remote: 0x90d377c4c8bd85fa expref: 5 pid: 157452 timeout: 5333 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5233.635479] LustreError: lustre-MDT0000: A client on nid 192.168.204.42@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5233.640809] LustreError: 157441:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.204.42@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000f2502be1/0xe48ab2792cd1c359 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.204.42@tcp remote: 0x90d377c4c8bd85fa expref: 6 pid: 157452 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5242.348462] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5242.358095] LustreError: Skipped 1 previous similar message [ 5245.695123] Lustre: server umount lustre-OST0000 complete [ 5256.452853] Lustre: server umount lustre-MDT0000 complete [ 5257.697357] LustreError: 157457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5257.723779] LustreError: 157457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 5261.237815] LustreError: 157437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787565538 with bad export cookie 16468171220946370206 [ 5261.249067] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5261.261819] LustreError: 157437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5267.735945] Lustre: server umount lustre-MDT0001 complete [ 5283.076626] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 5287.004668] Key type lgssc unregistered [ 5287.365153] LNet: 163567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5287.371289] LNetError: 163567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5287.390639] LNet: Removed LNI 192.168.204.142@tcp [ 5288.354863] Key type .llcrypt unregistered [ 5288.357528] Key type ._llcrypt unregistered [ 5313.387638] Key type ._llcrypt registered [ 5313.393913] Key type .llcrypt registered [ 5313.502347] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 5315.087679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5315.160808] alg: No test for adler32 (adler32-zlib) [ 5316.380814] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 5316.703158] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 5318.383215] Key type lgssc registered [ 5319.790134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5328.233916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5328.249947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5329.810419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5334.784098] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5340.142279] LustreError: 164984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5340.164474] LustreError: 164984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5343.959611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5344.338037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5348.984899] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5357.361199] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5363.991880] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5373.058648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5373.523157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5375.549874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5382.511459] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5393.872036] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5400.421523] Lustre: Failing over lustre-MDT0000 [ 5400.544708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5400.557455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5400.578420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5400.728869] Lustre: server umount lustre-MDT0000 complete [ 5402.156134] LustreError: 164981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5404.129717] LustreError: 164985:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5404.135563] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5409.255548] LustreError: 164979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5409.279516] LustreError: 164979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5413.225231] LustreError: 164964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.42@tcp arrived at 1787565690 with bad export cookie 17930429923194352809 [ 5413.246184] LustreError: 164964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5414.371404] LustreError: 164980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5414.389869] LustreError: 164980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5420.513430] Lustre: 164293:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787565681/real 1787565681] req@00000000f4d6fb2b x1874398380323200/t0(0) o400->MGC192.168.204.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787565697 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5420.536392] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5421.322899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5425.634469] LustreError: 164981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5425.652045] LustreError: 164981:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5430.760903] LustreError: 164292:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002e9ec2fd x1874398380329088/t0(0) o250->MGC192.168.204.142@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5431.171857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5431.248113] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5436.386037] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5436.425665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5436.880475] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5440.113993] Lustre: lustre-MDT0000: Denying connection for new client 15e7610d-cf85-4bed-9abb-460af39b1728 (at 192.168.204.42@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 5450.208508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5450.220851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5450.243966] Lustre: Skipped 1 previous similar message [ 5450.256873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5451.754618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5451.777130] Lustre: Skipped 1 previous similar message [ 5454.942422] Lustre: server umount lustre-OST0000 complete [ 5459.873641] LustreError: 168727:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5459.887575] Lustre: 168076:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5459.895453] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5459.922401] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5459.949823] LustreError: 168076:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000de5ba1f1 x1874398380344576/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5459.994301] LustreError: 168076:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5460.015661] LustreError: 168076:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5460.036337] Lustre: lustre-MDT0000: Recovery over after 0:24, of 2 clients 0 recovered and 2 were evicted. [ 5460.668932] Lustre: server umount lustre-MDT0000 complete [ 5461.986329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5462.000975] Lustre: Skipped 1 previous similar message [ 5462.014896] LustreError: 164979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5462.038622] LustreError: 164979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5464.774327] LustreError: 164964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787565742 with bad export cookie 17930429923194352907 [ 5464.778740] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5464.783095] LustreError: 164964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5467.399913] Lustre: server umount lustre-MDT0001 complete [ 5478.507812] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 5481.023483] Key type lgssc unregistered [ 5481.300622] LNet: 169482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5481.311523] LNetError: 169482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5481.324744] LNet: Removed LNI 192.168.204.142@tcp [ 5482.189278] Key type .llcrypt unregistered [ 5482.193566] Key type ._llcrypt unregistered [ 5507.961946] Key type ._llcrypt registered [ 5507.964870] Key type .llcrypt registered [ 5508.059341] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 5509.205196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5509.294898] alg: No test for adler32 (adler32-zlib) [ 5510.345646] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 5510.606051] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 5512.295195] Key type lgssc registered [ 5513.731250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5521.208985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5521.235610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5522.666170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5527.005672] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5533.161838] LustreError: 170897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5533.184680] LustreError: 170897:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5536.872085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5537.379452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5542.648245] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5554.187633] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5564.598526] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5575.299075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5575.921521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5583.251218] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5591.992798] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5593.348949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5603.828646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5603.829043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.843839] LustreError: Skipped 1 previous similar message [ 5603.859409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5608.541467] Lustre: server umount lustre-OST0000 complete [ 5614.058027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5614.074034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5614.081670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5614.097922] Lustre: Skipped 2 previous similar messages [ 5614.124105] Lustre: Skipped 2 previous similar messages [ 5618.719578] Lustre: server umount lustre-MDT0000 complete [ 5619.170804] LustreError: 170893:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5619.187721] LustreError: 170893:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5623.090215] LustreError: 173680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787565900 with bad export cookie 5130050906585343700 [ 5623.094154] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5623.098096] LustreError: 173680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5623.446844] Lustre: server umount lustre-MDT0001 complete [ 5635.150054] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 5638.447825] Key type lgssc unregistered [ 5638.917204] LNet: 174436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5638.928107] LNetError: 174436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5638.950976] LNet: Removed LNI 192.168.204.142@tcp [ 5640.103826] Key type .llcrypt unregistered [ 5640.106950] Key type ._llcrypt unregistered [ 5678.712608] Key type ._llcrypt registered [ 5678.715280] Key type .llcrypt registered [ 5678.806597] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 5680.110840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5680.226714] alg: No test for adler32 (adler32-zlib) [ 5681.310978] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 5681.556568] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 5683.232864] Key type lgssc registered [ 5684.424932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5693.929138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5693.964794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5695.693615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5701.585180] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5706.222815] LustreError: 176142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5706.254871] LustreError: 176142:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5711.336337] LustreError: 176141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5711.823096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5712.303730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5717.031265] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5725.696926] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5732.549145] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5740.111780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5740.512920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5741.573731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5747.476817] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5756.101416] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5767.145941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5767.153960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5767.156153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5767.176394] Lustre: Skipped 1 previous similar message [ 5772.260340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5772.282483] Lustre: Skipped 1 previous similar message [ 5777.382865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5777.388761] Lustre: Skipped 2 previous similar messages [ 5778.400434] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5778.835742] Lustre: server umount lustre-OST0000 complete [ 5784.044149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5784.065813] LustreError: Skipped 1 previous similar message [ 5784.081391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5784.108781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5787.641709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5789.852565] Lustre: server umount lustre-MDT0000 complete [ 5792.738239] LustreError: 176141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5792.753924] LustreError: 176141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5794.658032] LustreError: 176123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787566072 with bad export cookie 2891926013531474815 [ 5794.667794] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5794.671246] LustreError: 176123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5795.090818] Lustre: server umount lustre-MDT0001 complete [ 5807.246822] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 5811.938269] Key type lgssc unregistered [ 5812.612533] LNet: 179680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5812.634057] LNetError: 179680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5813.687273] LNet: Removed LNI 192.168.204.142@tcp [ 5815.010168] Key type .llcrypt unregistered [ 5815.014354] Key type ._llcrypt unregistered [ 5842.455322] Key type ._llcrypt registered [ 5842.458805] Key type .llcrypt registered [ 5842.622896] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 5843.736754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5843.815500] alg: No test for adler32 (adler32-zlib) [ 5844.993761] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 5845.300508] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 5847.009997] Key type lgssc registered [ 5848.472635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5861.527594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5861.565260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5863.076249] Lustre: *** cfs_fail_loc=135, val=0*** [ 5863.234138] LustreError: 181207:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5863.254867] LustreError: 181207:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.204.142@tcp: cfg command failed: rc = -2 [ 5863.274515] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5863.274515] [ 5863.290350] LustreError: MGC192.168.204.142@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5863.315581] LustreError: 181168:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5863.326904] LustreError: 181168:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 5863.344235] LustreError: 181168:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5863.354843] LustreError: 181168:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5863.477724] Lustre: server umount lustre-MDT0000 complete [ 5863.485918] LustreError: 181168:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5871.684343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5872.122411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5876.850428] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5882.337880] LustreError: 181579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5882.358196] LustreError: 181579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5886.159979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5886.570802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5891.653990] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 5900.289962] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5907.341388] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5910.694936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5910.718901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5916.082887] Lustre: server umount lustre-MDT0000 complete [ 5916.130771] LustreError: 182308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5916.148814] LustreError: 182308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5920.180992] LustreError: 181559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787566197 with bad export cookie 16128431543507762133 [ 5920.199777] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5920.200418] LustreError: 181559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5920.904423] Lustre: server umount lustre-MDT0001 complete [ 5935.625580] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 5939.224792] Key type lgssc unregistered [ 5939.616486] LNet: 183905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5939.623721] LNetError: 183905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5939.670891] LNet: Removed LNI 192.168.204.142@tcp [ 5940.922801] Key type .llcrypt unregistered [ 5940.925222] Key type ._llcrypt unregistered [ 5967.166216] Key type ._llcrypt registered [ 5967.168336] Key type .llcrypt registered [ 5967.325370] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 5967.640462] Key type .llcrypt unregistered [ 5967.644863] Key type ._llcrypt unregistered [ 5986.279512] Key type ._llcrypt registered [ 5986.282896] Key type .llcrypt registered [ 5986.388761] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 5988.032049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5988.063417] alg: No test for adler32 (adler32-zlib) [ 5989.196739] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 5989.472536] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 5991.183177] Key type lgssc registered [ 5992.601415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6001.052135] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6001.078710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6016.735452] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6021.856417] LustreError: 186045:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.142@tcp: failed processing log, type 4: rc = -110 [ 6048.543529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6056.889615] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6067.045026] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6075.934539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6076.466519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6076.552367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6081.507553] LustreError: 186071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6081.534957] LustreError: 186071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6082.209436] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6082.541260] LustreError: 187083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6087.655970] LustreError: 186071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6087.684544] LustreError: 186071:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6092.775797] LustreError: 186070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6092.790923] LustreError: 186070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6092.947448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6093.418034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6098.003472] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6106.382842] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6113.065783] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6116.276392] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6127.584484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6127.597756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6127.626353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6129.122205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6129.123043] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6129.133850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6129.135734] Lustre: Skipped 1 previous similar message [ 6133.355830] Lustre: server umount lustre-OST0000 complete [ 6139.366336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6139.367817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6139.401782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6139.412328] Lustre: Skipped 1 previous similar message [ 6144.415236] Lustre: server umount lustre-MDT0000 complete [ 6144.480854] LustreError: 187087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6144.498918] LustreError: 187087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6148.428780] LustreError: 186051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787566425 with bad export cookie 6627206221794846954 [ 6148.433614] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6148.435709] LustreError: 186051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6148.890922] Lustre: server umount lustre-MDT0001 complete [ 6164.164776] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6169.514454] Key type lgssc unregistered [ 6170.056798] LNet: 189968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6170.075814] LNetError: 189968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6170.106306] LNet: Removed LNI 192.168.204.142@tcp [ 6171.851526] Key type .llcrypt unregistered [ 6171.854827] Key type ._llcrypt unregistered [ 6202.189278] Key type ._llcrypt registered [ 6202.191936] Key type .llcrypt registered [ 6202.287235] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6203.726947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6203.771160] alg: No test for adler32 (adler32-zlib) [ 6204.824774] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 6205.074513] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6206.727228] Key type lgssc registered [ 6207.788070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6216.374508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6216.411810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6218.069359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6223.079941] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6228.485171] LustreError: 191381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6228.524985] LustreError: 191381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6233.570185] LustreError: 191382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6233.907885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6234.532409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6239.853194] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6249.272680] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6256.510492] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6263.791232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6264.196372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6265.209374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6270.454706] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6278.080209] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6282.724558] Lustre: Failing over lustre-MDT0000 [ 6283.022980] Lustre: server umount lustre-MDT0000 complete [ 6283.754925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6283.765789] LustreError: 191377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6283.776757] Lustre: Skipped 2 previous similar messages [ 6283.812193] LustreError: 191377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6288.869734] LustreError: 193826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6288.888515] LustreError: 193826:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6293.985092] LustreError: 191382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6294.001412] LustreError: 191382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6300.133194] Lustre: 190693:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787566561/real 1787566561] req@00000000e5c7ad81 x1874399311455744/t0(0) o400->MGC192.168.204.142@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787566577 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6300.160985] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6304.119744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6305.253095] LustreError: 192816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6305.273550] LustreError: 192816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6310.925103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6311.053593] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6311.978257] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6316.030830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6316.078799] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6316.141306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6317.257866] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6323.590350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6332.938792] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6346.727691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6346.734637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6346.750089] Lustre: Skipped 1 previous similar message [ 6351.853756] LustreError: 193206:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6351.896638] LustreError: 193206:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6352.045650] Lustre: server umount lustre-OST0000 complete [ 6356.967865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6356.971128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6357.004954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6357.021612] Lustre: Skipped 2 previous similar messages [ 6362.082249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6367.203347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6367.222763] Lustre: Skipped 2 previous similar messages [ 6371.295172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6371.730382] Lustre: server umount lustre-MDT0000 complete [ 6376.482721] LustreError: 194966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787566654 with bad export cookie 15776551109062859415 [ 6376.487420] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6376.505287] LustreError: 194966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6376.870352] Lustre: server umount lustre-MDT0001 complete [ 6388.954555] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6392.914990] Key type lgssc unregistered [ 6393.302487] LNet: 196294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6393.314321] LNetError: 196294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6393.358953] LNet: Removed LNI 192.168.204.142@tcp [ 6394.357941] Key type .llcrypt unregistered [ 6394.359951] Key type ._llcrypt unregistered [ 6420.089949] Key type ._llcrypt registered [ 6420.091957] Key type .llcrypt registered [ 6420.220693] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6421.347047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6421.384931] alg: No test for adler32 (adler32-zlib) [ 6422.529299] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 6422.801648] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6424.519324] Key type lgssc registered [ 6425.762723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6434.579490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6434.626103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6436.120070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6441.037247] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6445.538339] LustreError: 197707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6445.552729] LustreError: 197707:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6450.384381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6450.758503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6457.121177] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6466.140803] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6472.790610] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6481.049519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6481.446965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6486.518823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6487.826896] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6496.657364] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6500.529156] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6505.277313] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6522.569438] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6531.554877] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6531.555318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6531.560720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6531.606394] Lustre: Skipped 1 previous similar message [ 6534.617941] Lustre: server umount lustre-OST0000 complete [ 6541.799487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6541.825889] Lustre: Skipped 1 previous similar message [ 6541.830669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6541.843029] Lustre: Skipped 2 previous similar messages [ 6545.505171] Lustre: server umount lustre-MDT0000 complete [ 6546.931251] LustreError: 197703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6546.960092] LustreError: 197703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6549.707966] LustreError: 199554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787566827 with bad export cookie 3512233269253088700 [ 6549.709061] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6549.723250] LustreError: 199554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6550.093845] Lustre: server umount lustre-MDT0001 complete [ 6567.295447] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6570.738116] Key type lgssc unregistered [ 6571.118078] LNet: 201590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6571.132712] LNetError: 201590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6571.152120] LNet: Removed LNI 192.168.204.142@tcp [ 6572.202829] Key type .llcrypt unregistered [ 6572.206565] Key type ._llcrypt unregistered [ 6595.281404] Key type ._llcrypt registered [ 6595.285227] Key type .llcrypt registered [ 6595.431384] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6596.516043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6596.746727] alg: No test for adler32 (adler32-zlib) [ 6597.869720] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 6598.140753] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6599.815235] Key type lgssc registered [ 6601.365851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6609.822260] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6609.843839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6611.260409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6616.260463] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6620.661529] LustreError: 202857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6620.673620] LustreError: 202857:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6625.356432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6625.634602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6630.885450] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6638.928531] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6644.871376] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6653.722297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6654.272233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6656.307724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6661.653151] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6672.108949] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6679.242603] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6692.057337] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6701.093481] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6706.871213] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6717.923818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6717.935074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6717.950824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6722.016450] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6722.021903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6722.036283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6722.075691] Lustre: Skipped 1 previous similar message [ 6724.018355] Lustre: server umount lustre-OST0000 complete [ 6728.164654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6728.176514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6728.194548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6732.256076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6732.257453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6734.105782] Lustre: server umount lustre-MDT0000 complete [ 6737.380403] LustreError: 202851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6737.414358] LustreError: 202851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6738.292644] LustreError: 202838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787567015 with bad export cookie 15964809814835897334 [ 6738.301477] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6738.308431] LustreError: 202838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6738.647409] Lustre: server umount lustre-MDT0001 complete [ 6750.230656] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6753.255468] Key type lgssc unregistered [ 6753.509805] LNet: 207789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6753.518610] LNetError: 207789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6754.598738] LNet: Removed LNI 192.168.204.142@tcp [ 6755.422703] Key type .llcrypt unregistered [ 6755.425376] Key type ._llcrypt unregistered [ 6780.088123] Key type ._llcrypt registered [ 6780.092404] Key type .llcrypt registered [ 6780.238717] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6781.844664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6781.873924] alg: No test for adler32 (adler32-zlib) [ 6782.930307] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 6783.162859] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6784.879204] Key type lgssc registered [ 6786.189135] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6795.228753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6795.248555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6796.743853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6802.222958] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6807.017603] LustreError: 209201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.052798] LustreError: 209201:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6811.796242] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6812.165722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6817.816675] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6827.424669] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6836.225274] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6846.761689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6847.130314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6855.750405] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6865.066713] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6866.617701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6887.394162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6887.407283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6887.410510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6887.420433] Lustre: Skipped 1 previous similar message [ 6890.231249] Lustre: server umount lustre-OST0000 complete [ 6894.052232] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6894.059460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6894.083252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6894.088598] Lustre: Skipped 1 previous similar message [ 6897.647304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6897.650758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6899.881166] Lustre: server umount lustre-MDT0000 complete [ 6902.754444] LustreError: 209202:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6902.771831] LustreError: 209202:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6903.365103] LustreError: 209182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787567180 with bad export cookie 976991384306284438 [ 6903.379207] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6903.713790] Lustre: server umount lustre-MDT0001 complete [ 6913.815350] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 6916.627131] Key type lgssc unregistered [ 6917.078605] LNet: 212901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6917.097328] LNetError: 212901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6917.116142] LNet: Removed LNI 192.168.204.142@tcp [ 6917.990424] Key type .llcrypt unregistered [ 6917.996313] Key type ._llcrypt unregistered [ 6936.115554] Key type ._llcrypt registered [ 6936.118771] Key type .llcrypt registered [ 6936.213705] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 6936.984651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6937.026027] alg: No test for adler32 (adler32-zlib) [ 6938.065924] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 6938.251769] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 6939.903192] Key type lgssc registered [ 6940.903776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6949.980650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6949.998867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6951.499745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6956.507966] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6961.633561] LustreError: 214167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6961.651696] LustreError: 214167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6966.457394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.828636] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6971.991890] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 6980.657625] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6986.969572] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6993.416971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6993.804699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6996.921571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7000.406591] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7008.248694] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7027.685422] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7027.690242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7027.698957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7027.719037] Lustre: Skipped 1 previous similar message [ 7029.475619] Lustre: server umount lustre-OST0000 complete [ 7037.925256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7037.939100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7037.948323] Lustre: Skipped 2 previous similar messages [ 7043.045075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7043.054533] Lustre: Skipped 1 previous similar message [ 7048.161220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7048.162334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7048.199655] Lustre: Skipped 1 previous similar message [ 7048.358831] Lustre: server umount lustre-MDT0000 complete [ 7052.942697] LustreError: 214149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787567330 with bad export cookie 12633668129968190203 [ 7052.953798] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7053.445629] Lustre: server umount lustre-MDT0001 complete [ 7065.824963] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 7069.002224] Key type lgssc unregistered [ 7069.399037] LNet: 217858:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7069.406593] LNetError: 217858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7069.426243] LNet: Removed LNI 192.168.204.142@tcp [ 7070.417557] Key type .llcrypt unregistered [ 7070.424465] Key type ._llcrypt unregistered [ 7091.337278] Key type ._llcrypt registered [ 7091.341124] Key type .llcrypt registered [ 7091.456199] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7092.369923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7092.411705] alg: No test for adler32 (adler32-zlib) [ 7093.460519] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 7093.670773] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 7095.343206] Key type lgssc registered [ 7096.855353] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7106.459642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7106.486314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7108.090648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7113.853475] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7118.308171] LustreError: 219130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7118.347649] LustreError: 219130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7123.436627] LustreError: 219131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7124.368035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7124.823037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7130.036833] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7139.327875] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7145.912968] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7154.404234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7154.851915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7162.707262] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7173.135550] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7178.424149] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7188.569267] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7188.770864] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7189.015439] LustreError: 220953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.42@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7189.261114] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7197.169811] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7205.552752] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7207.497735] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7219.791441] Lustre: Permanently deactivating lustre-OST0001 [ 7219.825410] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7229.518463] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7237.306039] Lustre: Permanently reactivating lustre-OST0001 [ 7237.336798] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7237.346285] Lustre: Skipped 2 previous similar messages [ 7247.486565] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7247.499112] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7247.512263] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7247.526808] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7247.564177] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7251.443450] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7251.463647] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7251.473227] Lustre: Skipped 2 previous similar messages [ 7251.503940] Lustre: Skipped 1 previous similar message [ 7256.241692] Lustre: server umount lustre-OST0001 complete [ 7256.552975] LustreError: 220952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7256.573780] LustreError: 220952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7261.667933] LustreError: 220952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7261.668350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7261.677856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7261.707907] LustreError: 220952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7261.720428] Lustre: Skipped 1 previous similar message [ 7266.788701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7266.800030] Lustre: Skipped 1 previous similar message [ 7267.563786] Lustre: server umount lustre-OST0000 complete [ 7273.441037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7273.452092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7273.476788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7278.774142] Lustre: server umount lustre-MDT0000 complete [ 7282.154334] LustreError: 219858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7282.183349] LustreError: 219858:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7283.227822] LustreError: 219111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787567560 with bad export cookie 5403064434961590728 [ 7283.252949] LustreError: 219111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7283.258806] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7283.931200] Lustre: server umount lustre-MDT0001 complete [ 7296.655691] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 7300.151452] Key type lgssc unregistered [ 7300.419039] LNet: 224213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7300.428459] LNetError: 224213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7300.447425] LNet: Removed LNI 192.168.204.142@tcp [ 7301.477164] Key type .llcrypt unregistered [ 7301.482954] Key type ._llcrypt unregistered [ 7347.787901] Key type ._llcrypt registered [ 7347.796403] Key type .llcrypt registered [ 7347.963836] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7349.765969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7349.869072] alg: No test for adler32 (adler32-zlib) [ 7351.051956] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 7351.381593] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 7353.175355] Key type lgssc registered [ 7355.261982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7366.334353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7366.380101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7367.891411] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7367.932758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7368.185519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7373.826359] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7384.097873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7384.206562] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7384.255279] Lustre: 226422:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7384.487318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7389.388397] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7397.282386] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7403.239652] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7411.506806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7411.821684] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7412.151957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7419.246156] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7420.080859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7428.654190] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7432.574829] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7436.101967] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7442.024069] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7448.094413] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7456.655736] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7476.591928] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7476.602610] Lustre: Skipped 2 previous similar messages [ 7518.994505] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7519.006170] Lustre: Skipped 4 previous similar messages [ 7584.821221] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7584.832266] Lustre: Skipped 7 previous similar messages [ 7609.316145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7609.325535] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7609.342214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7609.829488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7609.853196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7610.657791] Lustre: server umount lustre-OST0000 complete [ 7620.069118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7620.078006] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7620.083159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7620.097718] Lustre: Skipped 1 previous similar message [ 7621.749186] Lustre: server umount lustre-MDT0000 complete [ 7626.254405] LustreError: 226412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787567903 with bad export cookie 17404256625290817226 [ 7626.260143] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7626.269350] LustreError: 226412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7626.682799] Lustre: server umount lustre-MDT0001 complete [ 7639.625965] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 7642.910678] Key type lgssc unregistered [ 7643.303834] LNet: 231234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7643.314330] LNetError: 231234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7643.334742] LNet: Removed LNI 192.168.204.142@tcp [ 7644.293680] Key type .llcrypt unregistered [ 7644.295761] Key type ._llcrypt unregistered [ 7672.596497] Key type ._llcrypt registered [ 7672.598720] Key type .llcrypt registered [ 7672.718120] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7673.770061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7673.925428] alg: No test for adler32 (adler32-zlib) [ 7674.936211] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 7675.229961] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 7676.959179] Key type lgssc registered [ 7678.226786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7686.294637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7686.327487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7687.777757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7692.531031] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7697.890238] LustreError: 232651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7697.911119] LustreError: 232651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7701.593916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7701.993471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7706.926905] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7714.980306] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7721.402754] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7728.454655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7728.959379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7733.045934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7736.657538] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_default_debug -1 all [ 7747.003605] Lustre: DEBUG MARKER: oleg442-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7763.938125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7763.947122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7763.962164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7768.034700] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7768.041171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7768.044138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7769.285266] Lustre: server umount lustre-OST0000 complete [ 7778.276982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7778.302056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7778.315924] Lustre: Skipped 2 previous similar messages [ 7780.240239] Lustre: server umount lustre-MDT0000 complete [ 7783.404900] LustreError: 232650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7785.091241] LustreError: 235230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787568062 with bad export cookie 1802095538688294169 [ 7785.108141] LustreError: MGC192.168.204.142@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7785.113586] LustreError: 235230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7785.570850] Lustre: server umount lustre-MDT0001 complete [ 7801.749188] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 7806.200656] Key type lgssc unregistered [ 7806.460137] LNet: 236345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7806.483601] LNetError: 236345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7806.519864] LNet: Removed LNI 192.168.204.142@tcp [ 7807.749885] Key type .llcrypt unregistered [ 7807.751826] Key type ._llcrypt unregistered [ 7832.445370] Key type ._llcrypt registered [ 7832.449405] Key type .llcrypt registered [ 7832.570475] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 7833.792800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7833.813258] alg: No test for adler32 (adler32-zlib) [ 7834.919910] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 7835.231845] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 7836.887240] Key type lgssc registered [ 7838.041506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8109.231801] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 8112.076869] Key type lgssc unregistered [ 8112.420260] LNet: 238368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8112.427186] LNetError: 238368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8112.446602] LNet: Removed LNI 192.168.204.142@tcp [ 8113.294840] Key type .llcrypt unregistered [ 8113.297119] Key type ._llcrypt unregistered [ 8142.485343] Key type ._llcrypt registered [ 8142.487968] Key type .llcrypt registered [ 8142.671683] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing set_hostid [ 8160.843340] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing load_modules_local [ 8162.291962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8162.365931] alg: No test for adler32 (adler32-zlib) [ 8163.425593] Lustre: Lustre: Build Version: 2.17.57_82_g4029ffa [ 8163.668568] LNet: Added LNI 192.168.204.142@tcp [8/256/0/180] [ 8165.351187] Key type lgssc registered [ 8166.974549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8205.017696] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7887 sec ========== 06:48:00 (1787568480) [ 8207.563842] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:48:02 (1787568482) === [ 8212.183105] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:48:07 (1787568487) === [ 8263.483898] Lustre: DEBUG MARKER: oleg442-server.virtnet: executing unload_modules_local [ 8266.237961] Key type lgssc unregistered [ 8266.531453] LNet: 243612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8266.542105] LNetError: 243612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8266.555804] LNet: Removed LNI 192.168.204.142@tcp [ 8267.273442] Key type .llcrypt unregistered [ 8267.276425] Key type ._llcrypt unregistered