[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 583377443 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2576MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002360] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.006763] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.010061] LSM: Security Framework initializing [ 0.011049] Yama: becoming mindful. [ 0.012033] SELinux: Initializing. [ 0.013063] *** VALIDATE selinux *** [ 0.022831] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027755] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028187] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030058] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031156] *** VALIDATE tmpfs *** [ 0.032539] *** VALIDATE proc *** [ 0.033000] *** VALIDATE cgroup *** [ 0.033017] *** VALIDATE cgroup2 *** [ 0.035003] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036186] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037016] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038039] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.043188] debug: unmapping init [mem 0xffffffff85a59000-0xffffffff85a60fff] [ 0.046267] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047828] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049030] ... version: 2 [ 0.050010] ... bit width: 48 [ 0.051015] ... generic registers: 4 [ 0.051969] ... value mask: 0000ffffffffffff [ 0.052018] ... max period: 00007fffffffffff [ 0.053017] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.056294] rcu: Hierarchical SRCU implementation. [ 0.059265] smp: Bringing up secondary CPUs ... [ 0.060774] x86: Booting SMP configuration: [ 0.061043] .... node #0, CPUs: #1 #2 #3 [ 0.065230] smp: Brought up 1 node, 4 CPUs [ 0.067035] smpboot: Max logical packages: 1 [ 0.068017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.231542] node 0 deferred pages initialised in 161ms [ 0.233144] devtmpfs: initialized [ 0.235355] x86/mm: Memory block size: 128MB [ 0.240267] gcov: version magic: 0x41383552 [ 0.242298] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.243148] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.244363] pinctrl core: initialized pinctrl subsystem [ 0.245287] [ 0.245919] ************************************************************* [ 0.246016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.247015] ** ** [ 0.248075] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.249020] ** ** [ 0.250016] ** This means that this kernel is built to expose internal ** [ 0.251017] ** IOMMU data structures, which may compromise security on ** [ 0.252019] ** your system. ** [ 0.253016] ** ** [ 0.254018] ** If you see this message and you are not debugging the ** [ 0.255019] ** kernel, report this immediately to your vendor! ** [ 0.256017] ** ** [ 0.257016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.258016] ************************************************************* [ 0.260000] NET: Registered protocol family 16 [ 0.261702] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.262079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.263073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.264876] cpuidle: using governor menu [ 0.266287] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.267782] PCI: Using configuration type 1 for base access [ 0.268127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.277070] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.278040] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.282336] cryptd: max_cpu_qlen set to 1000 [ 0.284243] ACPI: Added _OSI(Module Device) [ 0.285000] ACPI: Added _OSI(Processor Device) [ 0.285017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.286000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.294153] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.303377] ACPI: Interpreter enabled [ 0.305121] ACPI: PM: (supports S0 S3 S4 S5) [ 0.306011] ACPI: Using IOAPIC for interrupt routing [ 0.307227] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.308487] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.319521] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.320055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.321023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.322108] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.324449] acpiphp: Slot [2] registered [ 0.325168] acpiphp: Slot [5] registered [ 0.326167] acpiphp: Slot [6] registered [ 0.327205] acpiphp: Slot [7] registered [ 0.328164] acpiphp: Slot [8] registered [ 0.329280] acpiphp: Slot [9] registered [ 0.330131] acpiphp: Slot [10] registered [ 0.331173] acpiphp: Slot [3] registered [ 0.332140] acpiphp: Slot [4] registered [ 0.333123] acpiphp: Slot [11] registered [ 0.334232] acpiphp: Slot [12] registered [ 0.335097] acpiphp: Slot [13] registered [ 0.336202] acpiphp: Slot [14] registered [ 0.337083] acpiphp: Slot [15] registered [ 0.338128] acpiphp: Slot [16] registered [ 0.339093] acpiphp: Slot [17] registered [ 0.340158] acpiphp: Slot [18] registered [ 0.341118] acpiphp: Slot [19] registered [ 0.342065] acpiphp: Slot [20] registered [ 0.343047] acpiphp: Slot [21] registered [ 0.344071] acpiphp: Slot [22] registered [ 0.345081] acpiphp: Slot [23] registered [ 0.346187] acpiphp: Slot [24] registered [ 0.347114] acpiphp: Slot [25] registered [ 0.348052] acpiphp: Slot [26] registered [ 0.349073] acpiphp: Slot [27] registered [ 0.350149] acpiphp: Slot [28] registered [ 0.351087] acpiphp: Slot [29] registered [ 0.352109] acpiphp: Slot [30] registered [ 0.353147] acpiphp: Slot [31] registered [ 0.354083] PCI host bridge to bus 0000:00 [ 0.355011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.356021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.357031] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.358032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.359029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.360034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.361220] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.363308] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.365661] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.372394] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.375064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.376024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.377025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.378184] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.380455] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.382066] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.383067] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.385062] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.388018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.396065] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.399019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405053] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.418023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.429023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.460022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.475981] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.484021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.492021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.515019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.524588] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.530024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.538022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.554024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.564619] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.570019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.574023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.590018] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.600167] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.605018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.611017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.625018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.636520] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.644029] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.650019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.663022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.673597] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.678503] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.681474] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.684412] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.686295] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.692119] iommu: Default domain type: Passthrough [ 0.694600] SCSI subsystem initialized [ 0.696179] ACPI: bus type USB registered [ 0.697110] usbcore: registered new interface driver usbfs [ 0.699184] usbcore: registered new interface driver hub [ 0.701157] usbcore: registered new device driver usb [ 0.703136] pps_core: LinuxPPS API ver. 1 registered [ 0.704009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.705000] PTP clock support registered [ 0.705199] EDAC MC: Ver: 3.0.0 [ 0.708146] PCI: Using ACPI for IRQ routing [ 0.709965] NetLabel: Initializing [ 0.712013] NetLabel: domain hash size = 128 [ 0.714012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.716105] NetLabel: unlabeled traffic allowed by default [ 0.719314] vgaarb: loaded [ 0.721297] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.722020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.729201] clocksource: Switched to clocksource kvm-clock [ 0.862495] VFS: Disk quotas dquot_6.6.0 [ 0.864263] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.867034] *** VALIDATE ramfs *** [ 0.868417] *** VALIDATE hugetlbfs *** [ 0.870273] pnp: PnP ACPI init [ 0.872772] pnp: PnP ACPI: found 6 devices [ 0.890904] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.894569] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.896662] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.898435] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.899925] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.901459] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.903416] NET: Registered protocol family 2 [ 0.905206] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.908815] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.911100] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.915613] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.919203] TCP: Hash tables configured (established 65536 bind 65536) [ 0.922112] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.925390] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.928390] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.931607] NET: Registered protocol family 1 [ 0.934247] RPC: Registered named UNIX socket transport module. [ 0.936564] RPC: Registered udp transport module. [ 0.938499] RPC: Registered tcp transport module. [ 0.940484] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.943386] NET: Registered protocol family 44 [ 0.945369] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.947747] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.950299] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.953163] PCI: CLS 0 bytes, default 64 [ 0.955685] Unpacking initramfs... [ 2.488804] debug: unmapping init [mem 0xffff9d3dfcc54000-0xffff9d3dfffbffff] [ 2.492254] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.494431] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.497270] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.038095] Initialise system trusted keyrings [ 3.039864] Key type blacklist registered [ 3.043048] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.052535] zbud: loaded [ 3.055697] *** VALIDATE nfs *** [ 3.056966] *** VALIDATE nfs4 *** [ 3.058498] pstore: using deflate compression [ 3.064929] Platform Keyring initialized [ 3.181902] NET: Registered protocol family 38 [ 3.183728] Key type asymmetric registered [ 3.185567] Asymmetric key parser 'x509' registered [ 3.187720] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.191519] io scheduler mq-deadline registered [ 3.193590] io scheduler kyber registered [ 3.195616] io scheduler bfq registered [ 3.197614] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.201693] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.205324] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.208575] ACPI: Power Button [PWRF] [ 3.218406] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.226816] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.247124] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.256908] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.297241] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.322969] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.349818] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.354666] Non-volatile memory driver v1.3 [ 3.356349] Linux agpgart interface v0.103 [ 3.390375] virtio_blk virtio1: [vda] 146960 512-byte logical blocks (75.2 MB/71.8 MiB) [ 3.394834] vda: detected capacity change from 0 to 75243520 [ 3.441274] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.444606] vdb: detected capacity change from 0 to 1073741824 [ 3.462436] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.466260] vdc: detected capacity change from 0 to 2621440000 [ 3.487573] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.490861] vdd: detected capacity change from 0 to 2621440000 [ 3.510832] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.514614] vde: detected capacity change from 0 to 4294967296 [ 3.540570] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.545166] vdf: detected capacity change from 0 to 4294967296 [ 3.555167] libphy: Fixed MDIO Bus: probed [ 3.561663] usbcore: registered new interface driver usbserial_generic [ 3.564662] usbserial: USB Serial support registered for generic [ 3.567647] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.572839] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.575088] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.577648] mousedev: PS/2 mouse device common for all mice [ 3.581580] rtc_cmos 00:05: RTC can wake from S4 [ 3.583406] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.589390] rtc_cmos 00:05: registered as rtc0 [ 3.597103] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.597557] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.602773] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.604161] intel_pstate: CPU model not supported [ 3.607823] hid: raw HID events driver (C) Jiri Kosina [ 3.608996] usbcore: registered new interface driver usbhid [ 3.609907] usbhid: USB HID core driver [ 3.610771] drop_monitor: Initializing network drop monitor service [ 3.611972] Initializing XFRM netlink socket [ 3.613130] NET: Registered protocol family 10 [ 3.614776] Segment Routing with IPv6 [ 3.615780] NET: Registered protocol family 17 [ 3.617357] mpls_gso: MPLS GSO support [ 3.623348] RAS: Correctable Errors collector initialized. [ 3.624889] AVX version of gcm_enc/dec engaged. [ 3.626232] AES CTR mode by8 optimization enabled [ 3.702122] sched_clock: Marking stable (3702105717, 0)->(4807439171, -1105333454) [ 3.705666] registered taskstats version 1 [ 3.707779] Loading compiled-in X.509 certificates [ 3.710358] zswap: loaded using pool lzo/zbud [ 3.742391] Key type big_key registered [ 3.755746] Key type encrypted registered [ 3.757641] ima: No TPM chip found, activating TPM-bypass! [ 3.759881] ima: Allocated hash algorithm: sha1 [ 3.761336] ima: No architecture policies found [ 3.763343] evm: Initialising EVM extended attributes: [ 3.765792] evm: security.selinux [ 3.766846] evm: security.ima [ 3.768226] evm: security.capability [ 3.769648] evm: HMAC attrs: 0x1 [ 3.772427] rtc_cmos 00:05: setting system clock to 2026-08-27 14:30:48 UTC (1787841048) [ 3.780347] debug: unmapping init [mem 0xffffffff86a03000-0xffffffff86bfffff] [ 3.784254] debug: unmapping init [mem 0xffffffff85782000-0xffffffff85a58fff] [ 3.790123] Write protecting the kernel read-only data: 28672k [ 3.794576] debug: unmapping init [mem 0xffffffff83e03000-0xffffffff83ffffff] [ 3.797532] debug: unmapping init [mem 0xffffffff84714000-0xffffffff847fffff] [ 3.836286] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.845521] systemd[1]: Detected virtualization kvm. [ 3.847315] systemd[1]: Detected architecture x86-64. [ 3.849352] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.880512] systemd[1]: No hostname configured. [ 3.882537] systemd[1]: Set hostname to . [ 3.884675] random: systemd: uninitialized urandom read (16 bytes read) [ 3.887398] systemd[1]: Initializing machine ID from random generator. [ 3.968688] random: ln: uninitialized urandom read (6 bytes read) [ 4.170858] random: systemd: uninitialized urandom read (16 bytes read) [ 4.201980] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.266852] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 4.285341] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.936679] device-mapper: uevent: version 1.0.3 [ 4.938803] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook...[ 5.272840] random: fast init done [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.714803] virtio_net virtio0 ens2: renamed from eth0 [ 5.825451] scsi host0: ata_piix [ 5.929319] scsi host1: ata_piix [ 5.931420] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.935041] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.423889] random: crng init done [ 10.425404] random: 7 urandom warning(s) missed due to ratelimiting [ 12.000969] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.889526] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.861899] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.257054] SELinux: Disabled at runtime. [ 15.320808] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.329468] systemd[1]: Detected virtualization kvm. [ 15.331435] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.142192] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.145365] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.151995] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.156301] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.159729] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.172962] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.185032] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice[ 16.225882] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS . [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 17.327145] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.883081] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.891390] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.187929] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.216279] EDAC sbridge: Ver: 1.1.2 [ 20.573977] Key type dns_resolver registered [ 21.107127] NFS: Registering the id_resolver key type [ 21.109214] Key type id_resolver registered [ 21.110602] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg160-server login: [ 65.508438] hrtimer: interrupt took 2650962 ns [ 88.754430] libcfs: loading out-of-tree module taints kernel. [ 88.785080] Key type ._llcrypt registered [ 88.786499] Key type .llcrypt registered [ 88.862245] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 114.332829] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 116.842200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 116.921810] alg: No test for adler32 (adler32-zlib) [ 119.330561] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 121.324440] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 123.288208] Key type lgssc registered [ 126.642064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.859898] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 187.935840] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 204.405820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 204.487202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.007597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 206.037443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 206.152580] Lustre: lustre-MDT0000: new disk, initializing [ 206.371951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 206.416631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 210.406564] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.083885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 225.246601] Lustre: 6527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 225.288308] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 225.293357] Lustre: Skipped 1 previous similar message [ 225.376769] Lustre: lustre-MDT0001: new disk, initializing [ 225.441821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 225.488970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 225.518471] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 230.917928] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.477428] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 247.184309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 247.546630] Lustre: lustre-OST0000: new disk, initializing [ 247.550354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 247.555887] Lustre: 8464:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 247.649573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 254.786535] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 256.575686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 256.593259] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 256.733606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 271.191903] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 271.412519] Lustre: lustre-OST0001: new disk, initializing [ 271.417867] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 271.428924] Lustre: 9537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 271.535736] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 278.935440] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 280.648175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 280.662108] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 280.798439] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 291.199497] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 302.600057] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 309.410961] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing check_logdir /tmp/testlogs/ [ 314.758252] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing yml_node [ 319.662784] Lustre: DEBUG MARKER: Client: 2.17.57.98 [ 322.219351] Lustre: DEBUG MARKER: MDS: 2.17.57.98 [ 325.129757] Lustre: DEBUG MARKER: OSS: 2.17.57.98 [ 327.104456] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 10:36:10 EDT 2026 [ 345.241586] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 346.674661] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 362.977576] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.978166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 363.010986] Lustre: Skipped 3 previous similar messages [ 363.025304] Lustre: Skipped 3 previous similar messages [ 368.097576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 368.107883] Lustre: Skipped 3 previous similar messages [ 368.699680] Lustre: server umount lustre-MDT0000 complete [ 371.901993] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841416 with bad export cookie 17171598867000832497 [ 371.902425] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 371.927075] LustreError: 6518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 373.219421] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 373.223733] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 373.228403] Lustre: Skipped 1 previous similar message [ 373.236405] Lustre: Skipped 1 previous similar message [ 376.225069] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 376.229347] Lustre: Skipped 1 previous similar message [ 378.447767] Lustre: server umount lustre-MDT0001 complete [ 382.681637] Lustre: server umount lustre-OST0000 complete [ 387.896817] Lustre: server umount lustre-OST0001 complete [ 397.668825] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 409.054866] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 456.255059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.693776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 456.741491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 456.851582] Lustre: lustre-MDT0000: new disk, initializing [ 457.008272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 457.033616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 461.469958] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 471.676030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 471.905269] Lustre: 15096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 471.943662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 471.945929] Lustre: Skipped 1 previous similar message [ 472.028579] Lustre: lustre-MDT0001: new disk, initializing [ 472.079754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 472.100702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 472.109922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 475.900385] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 488.141441] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 495.792660] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 505.333875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 505.682074] Lustre: lustre-OST0000: new disk, initializing [ 505.688334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 505.701309] Lustre: 17068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 505.879636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 507.163264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 507.178179] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 507.413562] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 514.088359] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 525.090498] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 530.180434] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 530.410616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 535.641601] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 535.891742] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 538.082832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.086287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 538.103742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 538.109393] Lustre: Skipped 1 previous similar message [ 544.413269] Lustre: server umount lustre-OST0000 complete [ 548.839350] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 548.852394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.881187] Lustre: Skipped 1 previous similar message [ 548.892716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 548.900342] Lustre: Skipped 2 previous similar messages [ 553.441952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.833239] Lustre: server umount lustre-MDT0000 complete [ 558.562319] LustreError: 15107:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.583069] LustreError: 15107:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 559.281304] LustreError: 17067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841604 with bad export cookie 17171598867000834030 [ 559.298803] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 559.301862] LustreError: 17067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 559.602280] Lustre: server umount lustre-MDT0001 complete [ 563.461218] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 10:40:06 (1787841606) [ 570.303421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 570.629202] LustreError: 19332:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 570.693233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 575.975772] LustreError: 19333:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 576.052662] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 581.089797] LustreError: 19332:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 584.174494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 588.723816] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 595.736251] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 601.488354] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 610.077311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 610.213208] LustreError: 21153:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 619.485417] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 627.081668] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 634.191944] Lustre: server umount lustre-OST0000 complete [ 640.993261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 640.999796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 641.014336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 641.014344] Lustre: Skipped 2 previous similar messages [ 641.046459] Lustre: Skipped 1 previous similar message [ 649.380956] Lustre: server umount lustre-MDT0000 complete [ 651.233421] LustreError: 20060:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 651.256220] LustreError: 20060:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 652.128298] Lustre: 3660:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841680/real 1787841680] req@ffff9d3d43d6e940 x1874687338654976/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787841696 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'osp-pre-0-1.0' uid:0 gid:0 projid:4294967295 [ 654.249435] LustreError: 19314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841698 with bad export cookie 17171598867000834912 [ 654.293975] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 654.800307] Lustre: server umount lustre-MDT0001 complete [ 670.793700] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 674.169387] Key type lgssc unregistered [ 674.858296] LNet: 22871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.921216] LNetError: 22871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 674.980433] LNet: Removed LNI 192.168.201.160@tcp [ 676.376578] Key type .llcrypt unregistered [ 676.381776] Key type ._llcrypt unregistered [ 705.718615] Key type ._llcrypt registered [ 705.722549] Key type .llcrypt registered [ 705.922388] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 707.101177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 707.198236] alg: No test for adler32 (adler32-zlib) [ 708.614629] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 708.904223] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 710.560722] Key type lgssc registered [ 711.865946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 720.122342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 720.135894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.588580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 726.969293] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 732.131533] LustreError: 24283:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 732.141330] LustreError: 24283:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 736.628958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 737.302018] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 743.198976] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 750.517290] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 757.053163] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 764.315828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 764.828582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 766.922718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 772.273831] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 782.562415] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 802.785447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 802.787374] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 802.789918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 802.792428] LustreError: Skipped 1 previous similar message [ 802.826543] Lustre: Skipped 1 previous similar message [ 804.772986] Lustre: server umount lustre-OST0000 complete [ 813.026392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 813.041832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 813.055754] Lustre: Skipped 1 previous similar message [ 813.095938] Lustre: Skipped 2 previous similar messages [ 816.727223] Lustre: server umount lustre-MDT0000 complete [ 818.146852] LustreError: 25615:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 820.556795] LustreError: 27224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841865 with bad export cookie 8524779353746967984 [ 820.560445] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 820.577491] LustreError: 27224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 820.927896] Lustre: server umount lustre-MDT0001 complete [ 834.106499] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 837.308773] Key type lgssc unregistered [ 837.625342] LNet: 28266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.638211] LNetError: 28266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.700819] LNet: Removed LNI 192.168.201.160@tcp [ 839.925616] Key type .llcrypt unregistered [ 839.928396] Key type ._llcrypt unregistered [ 866.500795] Key type ._llcrypt registered [ 866.509241] Key type .llcrypt registered [ 866.699703] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 867.928612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 867.972094] alg: No test for adler32 (adler32-zlib) [ 869.301768] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 869.554848] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 871.288336] Key type lgssc registered [ 872.786202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 882.491079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 882.526421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 884.361297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 890.010502] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 894.947060] LustreError: 29680:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 894.980574] LustreError: 29680:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 899.330808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 899.815077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 905.511568] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 914.427974] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 921.259754] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 937.146529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 937.719360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 945.504275] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 953.964563] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 955.132369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 965.609771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 965.611290] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 965.622401] LustreError: Skipped 1 previous similar message [ 965.629953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 965.648800] Lustre: Skipped 1 previous similar message [ 965.700184] Lustre: Skipped 1 previous similar message [ 967.725322] Lustre: server umount lustre-OST0000 complete [ 975.851394] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.853817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 975.881516] Lustre: Skipped 1 previous similar message [ 975.892216] Lustre: Skipped 1 previous similar message [ 978.256031] Lustre: server umount lustre-MDT0000 complete [ 980.966334] LustreError: 29681:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 980.978054] LustreError: 29681:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 982.127155] LustreError: 29662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842026 with bad export cookie 9537218197474416560 [ 982.142693] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 982.143322] LustreError: 29662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 982.705299] Lustre: server umount lustre-MDT0001 complete [ 993.674946] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 996.109276] Key type lgssc unregistered [ 996.430464] LNet: 33563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 996.445137] LNetError: 33563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 996.479917] LNet: Removed LNI 192.168.201.160@tcp [ 997.536162] Key type .llcrypt unregistered [ 997.541421] Key type ._llcrypt unregistered [ 1024.045978] Key type ._llcrypt registered [ 1024.050273] Key type .llcrypt registered [ 1024.195883] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 1025.516239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1025.831482] alg: No test for adler32 (adler32-zlib) [ 1027.053564] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 1027.435812] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 1029.240155] Key type lgssc registered [ 1030.784445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1039.879722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1039.895675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1041.368955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1046.775289] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1051.617675] LustreError: 34979:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1051.631737] LustreError: 34979:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1055.982695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1056.379144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1061.730951] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1069.396726] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1076.175059] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1083.691292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1084.231776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1087.284478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1091.163945] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1100.310554] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1113.057860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1113.072823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1113.096676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1114.691485] Lustre: server umount lustre-OST0000 complete [ 1122.285167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.287902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1122.304681] Lustre: Skipped 1 previous similar message [ 1122.330117] Lustre: Skipped 1 previous similar message [ 1125.423867] Lustre: server umount lustre-MDT0000 complete [ 1127.397808] LustreError: 34975:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1127.409706] LustreError: 34975:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1129.721662] LustreError: 34959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842174 with bad export cookie 8270321839836828415 [ 1129.735220] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1129.746226] LustreError: 34959:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1140.428156] Lustre: server umount lustre-MDT0001 complete [ 1152.624069] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 1155.455375] Key type lgssc unregistered [ 1155.693859] LNet: 38516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.699405] LNetError: 38516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.710618] LNet: Removed LNI 192.168.201.160@tcp [ 1157.514200] Key type .llcrypt unregistered [ 1157.517278] Key type ._llcrypt unregistered [ 1182.127619] Key type ._llcrypt registered [ 1182.129934] Key type .llcrypt registered [ 1182.227284] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 1183.372705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1183.580469] alg: No test for adler32 (adler32-zlib) [ 1184.678520] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 1184.962366] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 1186.608224] Key type lgssc registered [ 1187.996404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1196.927969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1196.951122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.410696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1203.660414] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1208.811168] LustreError: 39932:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1208.829736] LustreError: 39932:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1213.923297] LustreError: 39933:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.976655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1214.607796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1219.202844] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1227.170813] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1233.042673] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1238.451773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1238.858590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1244.046471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1245.123428] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1253.009617] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1260.517687] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1260.530690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.549403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1264.612916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.616070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1264.640541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1264.668732] Lustre: Skipped 2 previous similar messages [ 1266.098077] Lustre: lustre-OST0000: Not available for connect from 192.168.201.60@tcp (stopping) [ 1269.743944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1269.766257] Lustre: Skipped 1 previous similar message [ 1274.337280] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1274.615496] Lustre: server umount lustre-OST0000 complete [ 1299.445075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1299.458220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1299.466476] Lustre: Skipped 1 previous similar message [ 1299.497306] Lustre: Skipped 2 previous similar messages [ 1310.690314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1310.697711] Lustre: Skipped 5 previous similar messages [ 1311.201302] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1311.545925] Lustre: server umount lustre-MDT0000 complete [ 1315.815334] LustreError: 39932:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1315.841705] LustreError: 39932:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1315.968274] LustreError: 39912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842360 with bad export cookie 13661222608081617105 [ 1315.974831] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1315.976275] LustreError: 39912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1323.046946] Lustre: server umount lustre-MDT0001 complete [ 1337.618308] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 1340.905865] Key type lgssc unregistered [ 1341.191591] LNet: 43649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1341.201559] LNetError: 43649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1341.225320] LNet: Removed LNI 192.168.201.160@tcp [ 1342.102697] Key type .llcrypt unregistered [ 1342.107690] Key type ._llcrypt unregistered [ 1371.137511] Key type ._llcrypt registered [ 1371.141553] Key type .llcrypt registered [ 1371.283227] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 1372.797236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1372.985955] alg: No test for adler32 (adler32-zlib) [ 1374.029258] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 1374.313396] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 1375.976198] Key type lgssc registered [ 1377.645714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1388.344620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1388.378557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.018735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1395.926861] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1400.292766] LustreError: 45067:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1400.316767] LustreError: 45067:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1405.422186] LustreError: 45066:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1407.421683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1408.215323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1414.034626] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1423.549082] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1431.576369] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1440.218204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1440.903539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.646303] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1457.861321] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1460.355157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1468.894313] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.60@tcp (stopping) [ 1469.414800] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1469.432368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1469.457418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1470.950420] LustreError: 45066:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1470.973804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1471.048464] Lustre: server umount lustre-MDT0000 complete [ 1473.963703] LustreError: 45062:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.041504] LustreError: 45062:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1476.041770] LustreError: 45047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842520 with bad export cookie 8448834773509407198 [ 1476.076560] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1476.076996] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.127687] Lustre: Skipped 1 previous similar message [ 1476.138535] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1479.078718] LustreError: 47156:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1479.094548] LustreError: 47156:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 1479.115476] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.60@tcp (stopping) [ 1484.217966] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.60@tcp (stopping) [ 1484.239515] Lustre: Skipped 1 previous similar message [ 1489.342557] LustreError: 45061:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.378882] LustreError: 45061:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 1490.402113] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1490.844505] Lustre: server umount lustre-MDT0001 complete [ 1507.518756] Lustre: server umount lustre-OST0000 complete [ 1525.535932] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 1528.241799] Key type lgssc unregistered [ 1528.475913] LNet: 48933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1528.485030] LNetError: 48933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1529.515784] LNet: Removed LNI 192.168.201.160@tcp [ 1530.611456] Key type .llcrypt unregistered [ 1530.615396] Key type ._llcrypt unregistered [ 1557.282928] Key type ._llcrypt registered [ 1557.285869] Key type .llcrypt registered [ 1557.424827] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 1558.661044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1558.690473] alg: No test for adler32 (adler32-zlib) [ 1559.959364] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 1560.246785] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 1561.928243] Key type lgssc registered [ 1563.315216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1572.179441] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1572.194238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1587.744496] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1592.864324] LustreError: 50324:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.160@tcp: failed processing log, type 4: rc = -110 [ 1619.616983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1626.327363] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1634.610607] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1800.693592] Lustre: server umount lustre-OST0000 complete [ 1815.816215] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 1818.907756] Key type lgssc unregistered [ 1819.207713] LNet: 52015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.214050] LNetError: 52015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1819.231692] LNet: Removed LNI 192.168.201.160@tcp [ 1820.126753] Key type .llcrypt unregistered [ 1820.131481] Key type ._llcrypt unregistered [ 1845.294267] Key type ._llcrypt registered [ 1845.299808] Key type .llcrypt registered [ 1845.391301] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 1846.482093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1846.662282] alg: No test for adler32 (adler32-zlib) [ 1847.917493] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 1848.167708] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 1849.832800] Key type lgssc registered [ 1851.265972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1859.716955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1859.737944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1861.167572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1866.334227] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1870.308296] LustreError: 53435:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1870.329887] LustreError: 53435:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 1874.742438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1875.100511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1879.636327] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1886.564418] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1892.452624] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1900.512585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1901.038803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1906.170590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1909.096409] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 1917.326793] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1921.507461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1921.513434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1921.524785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1924.586939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1924.599021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1925.778505] Lustre: server umount lustre-OST0000 complete [ 1931.237823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1931.243332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1931.253993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1931.260527] Lustre: Skipped 1 previous similar message [ 1934.822851] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1934.825277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1934.848375] Lustre: Skipped 1 previous similar message [ 1936.487031] Lustre: server umount lustre-MDT0000 complete [ 1939.943417] LustreError: 53435:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1940.402175] LustreError: 53414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842985 with bad export cookie 7332694118798064170 [ 1940.404060] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1940.430734] LustreError: 53414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1940.843760] Lustre: server umount lustre-MDT0001 complete [ 1953.977099] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 1957.938929] Key type lgssc unregistered [ 1958.384168] LNet: 56968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.423465] LNetError: 56968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.442531] LNet: Removed LNI 192.168.201.160@tcp [ 1959.569332] Key type .llcrypt unregistered [ 1959.577925] Key type ._llcrypt unregistered [ 1985.444722] Key type ._llcrypt registered [ 1985.451389] Key type .llcrypt registered [ 1985.592722] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 1987.001526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1987.062199] alg: No test for adler32 (adler32-zlib) [ 1988.124829] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 1988.434778] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 1990.097666] Key type lgssc registered [ 1991.692358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2000.040825] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2000.060405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2015.648477] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2020.768923] LustreError: 58357:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.160@tcp: failed processing log, type 4: rc = -110 [ 2047.392264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2054.028502] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2062.053460] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2070.172906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2070.694827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.889465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2076.137667] LustreError: 58383:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2076.157816] LustreError: 58383:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2076.314707] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2081.256176] LustreError: 58382:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2082.284347] LustreError: 58383:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2082.304652] LustreError: 58383:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2085.861947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.341534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2091.600534] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2099.516651] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2105.487553] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2111.974032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2111.974763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2111.975620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2112.007115] Lustre: Skipped 1 previous similar message [ 2113.694231] Lustre: server umount lustre-OST0000 complete [ 2136.545525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.550039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2136.559759] Lustre: Skipped 1 previous similar message [ 2136.566266] Lustre: Skipped 1 previous similar message [ 2141.651912] Lustre: server umount lustre-MDT0000 complete [ 2141.673910] LustreError: 59398:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.707417] LustreError: 59398:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2145.670217] LustreError: 58363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843190 with bad export cookie 18182679275935911583 [ 2145.688508] LustreError: 58363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2145.692185] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2152.165246] Lustre: server umount lustre-MDT0001 complete [ 2162.200592] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 2164.541825] Key type lgssc unregistered [ 2164.765548] LNet: 62103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2164.770449] LNetError: 62103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2164.785440] LNet: Removed LNI 192.168.201.160@tcp [ 2165.372213] Key type .llcrypt unregistered [ 2165.373709] Key type ._llcrypt unregistered [ 2190.754615] Key type ._llcrypt registered [ 2190.757113] Key type .llcrypt registered [ 2190.921390] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 2193.147716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2193.241291] alg: No test for adler32 (adler32-zlib) [ 2194.340514] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 2194.747631] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 2196.536161] Key type lgssc registered [ 2197.754042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2206.605841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2206.624390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.983811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2212.424519] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2217.459476] LustreError: 63516:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.486996] LustreError: 63516:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2221.475873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.826395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2226.473288] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2234.166403] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2239.927171] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2246.140356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2246.519514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.662914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2252.703840] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2260.471530] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2278.369472] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2278.397411] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.407567] Lustre: Skipped 1 previous similar message [ 2278.410450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2279.946987] Lustre: server umount lustre-OST0000 complete [ 2288.617383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2288.624947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2288.629084] Lustre: Skipped 1 previous similar message [ 2288.642197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2288.656102] Lustre: Skipped 2 previous similar messages [ 2288.690672] LustreError: Skipped 1 previous similar message [ 2290.682387] Lustre: server umount lustre-MDT0000 complete [ 2293.730524] LustreError: 64919:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2293.745781] LustreError: 64919:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2294.283819] LustreError: 63496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843339 with bad export cookie 2302797831101697789 [ 2294.287841] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2294.304980] LustreError: 63496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2294.693758] Lustre: server umount lustre-MDT0001 complete [ 2305.935801] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 2308.788672] Key type lgssc unregistered [ 2309.191920] LNet: 67074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.218920] LNetError: 67074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.243704] LNet: Removed LNI 192.168.201.160@tcp [ 2310.395725] Key type .llcrypt unregistered [ 2310.397360] Key type ._llcrypt unregistered [ 2339.093386] Key type ._llcrypt registered [ 2339.095964] Key type .llcrypt registered [ 2339.266989] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 2340.549583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2340.606594] alg: No test for adler32 (adler32-zlib) [ 2341.765798] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 2342.310347] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 2344.035153] Key type lgssc registered [ 2345.514474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2355.159513] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 11:09:58 (1787843398) [ 2361.869929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2361.889919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2363.515726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2368.429302] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2374.114890] LustreError: 68778:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.129556] LustreError: 68778:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2377.234613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2377.731561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2382.940788] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2390.360421] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2395.300479] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2402.156142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2402.550884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2408.691241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2410.109254] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2419.670768] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2425.253340] Lustre: Failing over lustre-MDT0000 [ 2425.571344] Lustre: server umount lustre-MDT0000 complete [ 2426.855830] LustreError: 69511:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2428.897897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2428.905431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.917099] LustreError: 68778:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.413951] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.424842] Lustre: Skipped 1 previous similar message [ 2431.917336] LustreError: 69511:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.942928] LustreError: 69511:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2434.651992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2434.832759] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2434.938540] Lustre: *** cfs_fail_loc=135, val=0*** [ 2435.077618] LustreError: 71862:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2435.086769] LustreError: 71862:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.160@tcp: cfg command failed: rc = -2 [ 2435.100731] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2435.100731] [ 2435.125622] LustreError: MGC192.168.201.160@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2435.154213] LustreError: 71851:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2435.170275] LustreError: 71851:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2435.197586] LustreError: 71851:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2435.206580] LustreError: 71851:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2435.244255] Lustre: server umount lustre-MDT0000 complete [ 2435.251502] LustreError: 71851:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2437.051452] LustreError: 69511:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2437.103659] LustreError: 69511:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2442.041854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2442.171470] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2442.191291] LustreError: 68770:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2442.422251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2442.478943] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2446.774806] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2447.272492] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2447.861067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2447.902143] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2447.955934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2452.965547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2452.976739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2452.981529] Lustre: Skipped 1 previous similar message [ 2455.079467] Lustre: server umount lustre-OST0000 complete [ 2463.201938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2463.207135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2463.214879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2463.214888] Lustre: Skipped 1 previous similar message [ 2463.222109] Lustre: Skipped 1 previous similar message [ 2464.786580] Lustre: server umount lustre-MDT0000 complete [ 2468.326849] LustreError: 69747:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.349863] LustreError: 69747:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 2468.894112] LustreError: 68758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843513 with bad export cookie 17192268913715869358 [ 2468.912472] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2468.914382] LustreError: 68758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2469.366962] Lustre: server umount lustre-MDT0001 complete [ 2480.611835] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 2483.859409] Key type lgssc unregistered [ 2484.400711] LNet: 73645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2484.408745] LNetError: 73645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2484.429880] LNet: Removed LNI 192.168.201.160@tcp [ 2485.765702] Key type .llcrypt unregistered [ 2485.768701] Key type ._llcrypt unregistered [ 2510.704150] Key type ._llcrypt registered [ 2510.711246] Key type .llcrypt registered [ 2510.819230] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 2512.190299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2512.204461] alg: No test for adler32 (adler32-zlib) [ 2513.157254] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 2513.342100] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 2514.977413] Key type lgssc registered [ 2516.507692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2525.274497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2525.291444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2526.656553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2531.593939] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2536.934518] LustreError: 75071:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.964375] LustreError: 75071:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2540.565871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2541.007709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2545.661793] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2553.295851] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2559.540256] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2567.131662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2567.608613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2572.742533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2575.453661] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2583.570293] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2593.118304] Lustre: Failing over lustre-MDT0000 [ 2593.652570] Lustre: server umount lustre-MDT0000 complete [ 2595.796561] LustreError: 75067:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.352723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2597.363081] Lustre: Skipped 1 previous similar message [ 2597.365649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2597.365833] LustreError: 75071:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.876468] LustreError: 76860:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2600.904439] LustreError: 76860:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2604.193803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.297973] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2604.473111] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2604.641754] LustreError: 78159:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2604.648141] LustreError: 78159:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.160@tcp: cfg command failed: rc = -9 [ 2604.657504] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2604.657504] [ 2604.665410] LustreError: MGC192.168.201.160@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2604.673523] LustreError: 78148:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2604.678458] LustreError: 78148:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2604.691755] LustreError: 78148:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2604.697427] LustreError: 78148:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2604.729597] Lustre: server umount lustre-MDT0000 complete [ 2604.734227] LustreError: 78148:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2605.996363] LustreError: 75068:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.015439] LustreError: 75068:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 2611.496380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.594871] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2611.604210] LustreError: 78506:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.201.160@tcp: namespace resource [0x65727473756c:0x0:0x0].0x0 (0000000055d73338) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 2611.617468] LustreError: 75063:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 2611.887328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.954591] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2612.099250] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2617.131927] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2617.330468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2617.366628] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2617.434339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2627.573936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2627.582901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2627.599290] Lustre: Skipped 2 previous similar messages [ 2627.626447] Lustre: Skipped 1 previous similar message [ 2631.550367] Lustre: server umount lustre-OST0000 complete [ 2637.793551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2637.796252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.812874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.821741] Lustre: Skipped 1 previous similar message [ 2637.839687] Lustre: Skipped 1 previous similar message [ 2641.830695] Lustre: server umount lustre-MDT0000 complete [ 2642.927223] LustreError: 75066:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2642.967054] LustreError: 75066:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 2645.696291] LustreError: 75052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843690 with bad export cookie 3869362554545778455 [ 2645.698732] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2645.706078] LustreError: 75052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2646.115562] Lustre: server umount lustre-MDT0001 complete [ 2657.971106] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 2661.251602] Key type lgssc unregistered [ 2661.808797] LNet: 79941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2661.829848] LNetError: 79941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2662.888174] LNet: Removed LNI 192.168.201.160@tcp [ 2663.842931] Key type .llcrypt unregistered [ 2663.846065] Key type ._llcrypt unregistered [ 2678.780414] Key type ._llcrypt registered [ 2678.782020] Key type .llcrypt registered [ 2678.885459] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 2679.297523] Key type .llcrypt unregistered [ 2679.299108] Key type ._llcrypt unregistered [ 2704.134041] Key type ._llcrypt registered [ 2704.138896] Key type .llcrypt registered [ 2704.333241] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 2705.927765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2705.991850] alg: No test for adler32 (adler32-zlib) [ 2707.096328] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 2707.388264] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 2709.120171] Key type lgssc registered [ 2710.469845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2718.079391] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2718.099936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2719.771448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2723.996924] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2729.957307] LustreError: 81848:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2729.979569] LustreError: 81848:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2733.531390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2733.857460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2738.350320] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2745.944479] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2752.098532] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2758.390963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2758.776943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2764.921434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2765.338757] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2772.887676] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2785.762274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2785.768204] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2785.782123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2785.796179] Lustre: Skipped 1 previous similar message [ 2785.815859] Lustre: Skipped 1 previous similar message [ 2788.247519] Lustre: server umount lustre-OST0000 complete [ 2795.489753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2795.513479] LustreError: Skipped 1 previous similar message [ 2795.528362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2795.548370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2799.754647] Lustre: server umount lustre-MDT0000 complete [ 2801.125113] LustreError: 81844:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.144334] LustreError: 81844:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2804.304782] LustreError: 81830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843849 with bad export cookie 16713947124080254462 [ 2804.332765] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2804.589759] Lustre: server umount lustre-MDT0001 complete [ 2817.670587] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 2820.667753] Key type lgssc unregistered [ 2820.960419] LNet: 85394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2820.964357] LNetError: 85394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2820.988362] LNet: Removed LNI 192.168.201.160@tcp [ 2822.127359] Key type .llcrypt unregistered [ 2822.129049] Key type ._llcrypt unregistered [ 2845.132952] Key type ._llcrypt registered [ 2845.136865] Key type .llcrypt registered [ 2845.277101] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 2846.370308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2846.571610] alg: No test for adler32 (adler32-zlib) [ 2847.630427] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 2847.886160] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 2849.545372] Key type lgssc registered [ 2850.680873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2858.696578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2858.747099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2860.223553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2864.751957] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2869.742599] LustreError: 86810:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.768057] LustreError: 86810:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 2873.866778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2874.223894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2878.982022] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2885.596356] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2890.956721] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2897.947082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2898.273555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2904.528591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2906.312951] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 2914.577331] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2923.666501] Lustre: server umount lustre-OST0000 complete [ 2930.148452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2930.158090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2930.158740] Lustre: Skipped 1 previous similar message [ 2930.169288] Lustre: Skipped 1 previous similar message [ 2935.266580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2935.276607] Lustre: Skipped 1 previous similar message [ 2938.422558] Lustre: server umount lustre-MDT0000 complete [ 2940.384421] Lustre: 86121:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787843969/real 1787843969] req@000000002f6ee7e1 x1874690196924672/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787843985 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2940.391526] LustreError: 86811:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2940.412895] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2940.444476] LustreError: 86811:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 2942.597478] LustreError: 86789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843987 with bad export cookie 993831022880695582 [ 2942.608260] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2942.615399] LustreError: 86789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2943.146725] Lustre: server umount lustre-MDT0001 complete [ 2955.237150] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 2958.676759] Key type lgssc unregistered [ 2958.971197] LNet: 90344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2958.977602] LNetError: 90344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2958.996236] LNet: Removed LNI 192.168.201.160@tcp [ 2959.759716] Key type .llcrypt unregistered [ 2959.764465] Key type ._llcrypt unregistered [ 2985.197394] Key type ._llcrypt registered [ 2985.199757] Key type .llcrypt registered [ 2985.372792] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 2987.061138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2987.105436] alg: No test for adler32 (adler32-zlib) [ 2988.347361] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 2988.706126] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 2990.432274] Key type lgssc registered [ 2991.946703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3000.965729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3001.030286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3002.689476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3008.193470] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3013.090422] LustreError: 91760:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3013.103763] LustreError: 91760:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3017.839986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3018.230110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3023.452797] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3032.130034] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3038.259589] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3046.137458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3046.684177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.738753] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3053.202960] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3061.579640] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3078.627270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3078.636443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3078.652638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3078.677890] Lustre: Skipped 1 previous similar message [ 3080.676604] Lustre: server umount lustre-OST0000 complete [ 3088.867077] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3088.873454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3088.876058] Lustre: Skipped 1 previous similar message [ 3088.883127] Lustre: Skipped 1 previous similar message [ 3091.704598] Lustre: server umount lustre-MDT0000 complete [ 3094.001638] LustreError: 91760:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3094.015200] LustreError: 91760:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3095.428092] LustreError: 91741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844140 with bad export cookie 1612475203949294127 [ 3095.429738] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3095.446814] LustreError: 91741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3095.925119] Lustre: server umount lustre-MDT0001 complete [ 3106.910683] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 3110.085820] Key type lgssc unregistered [ 3110.398989] LNet: 95317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3110.415658] LNetError: 95317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3111.466520] LNet: Removed LNI 192.168.201.160@tcp [ 3112.286564] Key type .llcrypt unregistered [ 3112.288607] Key type ._llcrypt unregistered [ 3137.662772] Key type ._llcrypt registered [ 3137.664829] Key type .llcrypt registered [ 3137.740368] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 3139.131894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3139.172676] alg: No test for adler32 (adler32-zlib) [ 3140.226502] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 3140.597879] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 3142.360195] Key type lgssc registered [ 3143.766962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3153.740324] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3153.787931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3169.376810] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3174.496514] LustreError: 96704:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.160@tcp: failed processing log, type 4: rc = -110 [ 3201.184219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3208.870160] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3218.249718] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3225.410746] Lustre: server umount lustre-OST0000 complete [ 3235.386223] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 11:24:38 (1787844278) [ 3242.363196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3242.905289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3247.878735] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3248.105949] LustreError: 98346:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3248.130581] LustreError: 98346:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3253.239339] LustreError: 98347:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3257.768988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3258.188313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3264.008090] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3273.222690] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3280.169597] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3287.972161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3288.573026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3295.502511] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3305.373372] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3313.207743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3328.486330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3328.500744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3328.506937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3328.518054] Lustre: Skipped 1 previous similar message [ 3333.387606] Lustre: server umount lustre-OST0000 complete [ 3338.728588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3338.746612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3338.755367] Lustre: Skipped 1 previous similar message [ 3338.778415] Lustre: Skipped 2 previous similar messages [ 3343.514128] Lustre: server umount lustre-MDT0000 complete [ 3343.851131] LustreError: 98346:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3343.880840] LustreError: 98346:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 3348.495175] LustreError: 98327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844393 with bad export cookie 14560016103506615655 [ 3348.512099] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3348.513981] LustreError: 98327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3349.100806] Lustre: server umount lustre-MDT0001 complete [ 3360.631295] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 3363.756622] Key type lgssc unregistered [ 3364.094522] LNet: 102066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3364.098035] LNetError: 102066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3364.117366] LNet: Removed LNI 192.168.201.160@tcp [ 3364.941552] Key type .llcrypt unregistered [ 3364.951429] Key type ._llcrypt unregistered [ 3410.080718] Key type ._llcrypt registered [ 3410.082666] Key type .llcrypt registered [ 3410.185563] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 3411.393969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3411.554441] alg: No test for adler32 (adler32-zlib) [ 3412.591937] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 3412.857671] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 3414.560455] Key type lgssc registered [ 3415.854779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3424.260895] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3424.306108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3425.735225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3431.587622] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3436.005857] LustreError: 104251:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3436.038075] LustreError: 104251:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 3441.134641] LustreError: 104252:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3441.525534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3442.155237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3447.933231] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3456.370085] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3463.359230] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3472.483450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3472.948878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3481.608719] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3490.636959] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3495.566932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3506.146801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3506.147173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3506.167589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3506.170330] Lustre: Skipped 1 previous similar message [ 3510.757598] Lustre: server umount lustre-OST0000 complete [ 3516.386923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3516.397030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3516.406447] Lustre: Skipped 2 previous similar messages [ 3516.418980] Lustre: Skipped 1 previous similar message [ 3521.396881] Lustre: server umount lustre-MDT0000 complete [ 3521.507740] LustreError: 104252:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3525.199262] LustreError: 104232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844569 with bad export cookie 2575974474423900710 [ 3525.203747] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3525.208364] LustreError: 104232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3525.548443] Lustre: server umount lustre-MDT0001 complete [ 3538.250359] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 3541.299312] Key type lgssc unregistered [ 3541.624619] LNet: 107813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3541.645699] LNetError: 107813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3541.661953] LNet: Removed LNI 192.168.201.160@tcp [ 3542.629927] Key type .llcrypt unregistered [ 3542.631961] Key type ._llcrypt unregistered [ 3565.004651] Key type ._llcrypt registered [ 3565.022993] Key type .llcrypt registered [ 3565.177318] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 3567.136098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3567.219136] alg: No test for adler32 (adler32-zlib) [ 3568.335355] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 3568.642197] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 3570.361063] Key type lgssc registered [ 3571.764757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3581.676543] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3581.699470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3597.282660] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3602.400312] LustreError: 109057:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.160@tcp: failed processing log, type 4: rc = -110 [ 3629.088281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3636.075302] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3643.345822] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3650.058106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3650.283440] LustreError: MGC192.168.201.160@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3650.293380] LustreError: 110082:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3650.299164] LustreError: 110082:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3650.304406] LustreError: 110082:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3650.381394] Lustre: server umount lustre-MDT0000 complete [ 3650.384432] LustreError: 110082:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3668.077577] Lustre: server umount lustre-OST0000 complete [ 3679.332660] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 3689.647939] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 3730.548651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3730.885738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3730.966253] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3731.088306] Lustre: lustre-MDT0000: new disk, initializing [ 3731.216506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3731.248079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3736.947677] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3748.883596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3749.109380] Lustre: 112873:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3749.210923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3749.214660] Lustre: Skipped 1 previous similar message [ 3749.287550] Lustre: lustre-MDT0001: new disk, initializing [ 3749.365719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3749.416920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3749.432147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3754.795236] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3765.023081] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3771.611631] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3780.878920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3781.504252] Lustre: lustre-OST0000: new disk, initializing [ 3781.509469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3781.515164] Lustre: 114840:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3781.765123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3788.939426] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3789.484625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3789.503073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3789.753509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3800.293771] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3806.311393] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3806.773430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3812.953442] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3813.390519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3820.519311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3820.523287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.533925] LustreError: Skipped 1 previous similar message [ 3820.538098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3820.581401] Lustre: Skipped 1 previous similar message [ 3822.452955] Lustre: server umount lustre-OST0000 complete [ 3830.755747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3830.770923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3830.781774] Lustre: Skipped 2 previous similar messages [ 3833.782581] Lustre: server umount lustre-MDT0000 complete [ 3838.622313] LustreError: 112865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844883 with bad export cookie 9177140003555696556 [ 3838.643797] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3839.124061] Lustre: server umount lustre-MDT0001 complete [ 3850.991322] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 11:34:53 (1787844893) [ 3855.001647] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3872.892602] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 3883.995028] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 3922.542163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3922.919443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3922.983812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3923.167919] Lustre: lustre-MDT0000: new disk, initializing [ 3923.225407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3923.264941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3928.663821] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3940.068175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3940.318922] Lustre: 119748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3940.489631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3940.519253] Lustre: Skipped 1 previous similar message [ 3940.689207] Lustre: lustre-MDT0001: new disk, initializing [ 3940.866880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3940.909301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3940.930330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3945.425193] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3957.947397] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3964.485861] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3972.103586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3972.518865] Lustre: lustre-OST0000: new disk, initializing [ 3972.522385] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3972.530817] Lustre: 121716:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3972.691074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3980.340128] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 3981.443490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3981.469359] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3981.676914] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3990.009429] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3994.556961] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3994.655116] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3999.670107] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3999.992906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4002.297068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4002.298394] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4002.307226] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4002.313445] Lustre: Skipped 2 previous similar messages [ 4007.398234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4007.406869] Lustre: Skipped 2 previous similar messages [ 4009.012986] Lustre: server umount lustre-OST0000 complete [ 4017.633912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4017.634955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4017.652073] Lustre: Skipped 1 previous similar message [ 4017.667363] Lustre: Skipped 1 previous similar message [ 4022.049826] Lustre: server umount lustre-MDT0000 complete [ 4022.754130] LustreError: 119754:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4022.771989] LustreError: 119754:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 4026.430810] LustreError: 119741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845071 with bad export cookie 9177140003555697438 [ 4026.440256] LustreError: 119741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4026.443728] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4026.852217] Lustre: server umount lustre-MDT0001 complete [ 4037.174832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4037.654609] LustreError: 123935:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4037.752351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4043.103813] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4043.235431] LustreError: 123936:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4047.355208] LustreError: 123935:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4051.230247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4057.838232] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4067.814932] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4075.280378] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4082.929434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4090.722840] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4099.097127] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4116.589988] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4122.595584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4122.610062] LustreError: Skipped 1 previous similar message [ 4122.620161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4122.633916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4127.590929] Lustre: server umount lustre-OST0000 complete [ 4133.345560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4133.363648] LustreError: Skipped 1 previous similar message [ 4133.371407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4133.392182] Lustre: Skipped 1 previous similar message [ 4138.979450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4138.985665] Lustre: Skipped 5 previous similar messages [ 4146.146777] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4146.389621] Lustre: server umount lustre-MDT0000 complete [ 4149.222819] LustreError: 123931:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4149.255310] LustreError: 123931:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4150.129791] LustreError: 123915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845194 with bad export cookie 9177140003555698313 [ 4150.132966] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4150.140529] LustreError: 123915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4150.443180] Lustre: server umount lustre-MDT0001 complete [ 4162.133542] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 4165.685380] Key type lgssc unregistered [ 4166.070208] LNet: 127589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4166.077435] LNetError: 127589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4167.143389] LNet: Removed LNI 192.168.201.160@tcp [ 4168.284421] Key type .llcrypt unregistered [ 4168.288132] Key type ._llcrypt unregistered [ 4187.933917] Key type ._llcrypt registered [ 4187.936622] Key type .llcrypt registered [ 4188.159848] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 4206.177394] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 4207.401529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4207.588717] alg: No test for adler32 (adler32-zlib) [ 4208.858271] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 4209.313646] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 4211.048179] Key type lgssc registered [ 4212.268669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4251.701097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4251.743753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4253.374590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4253.444826] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4253.566488] Lustre: lustre-MDT0000: new disk, initializing [ 4253.711018] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4253.756769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4258.497942] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4270.833737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4271.004916] Lustre: 130744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4271.061932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4271.070384] Lustre: Skipped 1 previous similar message [ 4271.170446] Lustre: lustre-MDT0001: new disk, initializing [ 4271.273465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4271.309539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4271.324157] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4276.902962] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4287.675875] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4293.103382] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4301.300465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4301.664410] Lustre: lustre-OST0000: new disk, initializing [ 4301.668052] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4301.677879] Lustre: 132713:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4301.779339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4307.523793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4307.551583] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4307.664849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4308.782946] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4319.283145] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4324.700356] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4324.941595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4330.595478] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4330.946507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4333.538108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4333.557278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4333.585836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4338.662746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4338.676874] Lustre: Skipped 1 previous similar message [ 4343.784250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4343.791208] Lustre: Skipped 1 previous similar message [ 4347.363324] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4347.800596] Lustre: server umount lustre-OST0000 complete [ 4353.508917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4353.517387] LustreError: Skipped 1 previous similar message [ 4353.527524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.552244] Lustre: Skipped 1 previous similar message [ 4353.571185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4358.522653] Lustre: server umount lustre-MDT0000 complete [ 4362.772857] LustreError: 130738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845407 with bad export cookie 838690655972417797 [ 4362.780682] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4362.788489] LustreError: 130738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4363.268816] Lustre: server umount lustre-MDT0001 complete [ 4373.964939] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 11:43:36 (1787845416) [ 4381.942889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4382.388027] LustreError: 135127:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4382.407163] LustreError: 135127:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4382.455681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4386.972318] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4387.813898] LustreError: 135128:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4392.936698] LustreError: 135127:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4397.333406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4398.018272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4405.402348] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4418.067102] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4424.448780] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4428.778338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4428.783704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4428.788273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4428.807570] Lustre: Skipped 2 previous similar messages [ 4428.841705] Lustre: Skipped 3 previous similar messages [ 4433.035781] Lustre: server umount lustre-MDT0000 complete [ 4433.891071] LustreError: 135128:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4433.909378] LustreError: 135128:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4436.258712] LustreError: 136698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845480 with bad export cookie 838690655972418686 [ 4436.271511] LustreError: 136698:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4436.277333] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4436.479702] Lustre: server umount lustre-MDT0001 complete [ 4446.740599] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 11:44:49 (1787845489) [ 4455.363989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4471.076295] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4476.194704] LustreError: 137544:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.160@tcp: failed processing log, type 4: rc = -110 [ 4501.984461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4509.605458] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4517.839823] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4520.677888] Lustre: server umount lustre-OST0000 complete [ 4530.968874] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 11:46:14 (1787845574) [ 4538.891291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4539.606673] LustreError: 138991:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4539.701125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4545.101313] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4550.116690] LustreError: 138991:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4550.131488] LustreError: 138991:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 4555.294264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4560.833409] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4569.500733] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4575.806236] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4582.746720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4583.199103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4583.214935] Lustre: Skipped 1 previous similar message [ 4590.950463] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4599.700598] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4617.185136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4617.193712] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4617.201162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4617.235962] Lustre: Skipped 1 previous similar message [ 4620.398452] Lustre: server umount lustre-MDT0000 complete [ 4622.305627] LustreError: 140192:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4622.326588] LustreError: 140192:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 4624.849273] LustreError: 138972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845669 with bad export cookie 838690655972419344 [ 4624.857415] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4624.865865] LustreError: 138972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4625.450474] Lustre: server umount lustre-MDT0001 complete [ 4639.733854] Lustre: server umount lustre-OST0000 complete [ 4649.421293] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 11:48:12 (1787845692) [ 4657.332670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4657.964568] LustreError: 142673:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4658.065805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4663.663700] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4674.681674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4681.295391] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4690.024717] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4697.316891] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4705.832091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4713.812865] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4723.512034] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4727.749215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4729.219920] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4729.558508] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4734.912280] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4735.101733] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4738.038242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4738.042918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4738.049132] Lustre: Skipped 1 previous similar message [ 4738.050608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4738.072033] Lustre: Skipped 3 previous similar messages [ 4744.091475] Lustre: server umount lustre-OST0000 complete [ 4751.841322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4751.856414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4751.880856] Lustre: Skipped 1 previous similar message [ 4755.002032] Lustre: server umount lustre-MDT0000 complete [ 4758.506269] LustreError: 142673:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4758.531081] LustreError: 142673:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 4760.158288] LustreError: 145986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845804 with bad export cookie 838690655972420422 [ 4760.177550] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4760.179484] LustreError: 145986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4761.013220] Lustre: server umount lustre-MDT0001 complete [ 4771.355653] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 11:50:14 (1787845814) [ 4778.247374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4793.953857] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4799.072493] LustreError: 146832:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.160@tcp: failed processing log, type 4: rc = -110 [ 4824.800425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4824.805766] Lustre: Skipped 2 previous similar messages [ 4832.879695] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4842.284956] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4850.559381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4851.272945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4856.866874] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4866.912840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4872.649718] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 4881.113558] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4888.345661] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4894.115922] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4894.316305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4899.219650] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4899.496595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4903.395503] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4903.398572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4903.403223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4903.403235] Lustre: Skipped 5 previous similar messages [ 4903.419987] Lustre: Skipped 2 previous similar messages [ 4908.446902] Lustre: server umount lustre-MDT0000 complete [ 4908.518662] LustreError: 146857:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4908.544265] LustreError: 146857:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 8 previous similar messages [ 4912.656304] LustreError: 146840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845957 with bad export cookie 838690655972421444 [ 4912.656304] LustreError: 146839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845957 with bad export cookie 838690655972421444 [ 4912.659915] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4912.674057] LustreError: 146840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4913.020914] Lustre: server umount lustre-MDT0001 complete [ 4928.047654] Lustre: server umount lustre-OST0000 complete [ 4937.137207] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 11:53:00 (1787845980) [ 4944.212398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4959.777264] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4964.898760] LustreError: 151030:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.160@tcp: failed processing log, type 4: rc = -110 [ 4997.802705] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5006.470222] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5014.667755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5014.927292] LustreError: 128965:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000dc735ea9 x1874691625224704/t0(0) o250->MGC192.168.201.160@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5015.579807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5020.979430] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5030.064792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5036.789476] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5045.072289] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5051.573589] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5059.493458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5059.728445] Lustre: lustre-OST0001: new disk, initializing [ 5059.730689] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5059.743370] Lustre: 153904:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5065.294000] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5065.301552] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5065.462512] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5067.353989] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5077.510478] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5083.943855] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5084.255283] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5089.721062] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5090.001879] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5092.329693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5092.334108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5092.340803] Lustre: Skipped 1 previous similar message [ 5092.344983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5092.344992] Lustre: Skipped 2 previous similar messages [ 5098.357964] Lustre: server umount lustre-OST0000 complete [ 5108.905704] Lustre: server umount lustre-OST0001 complete [ 5119.987628] Lustre: server umount lustre-MDT0000 complete [ 5124.129759] LustreError: 153903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846168 with bad export cookie 838690655972422396 [ 5124.137690] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5124.144196] LustreError: 153903:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5124.413875] Lustre: server umount lustre-MDT0001 complete [ 5154.975587] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 11:56:38 (1787846198) [ 5156.927461] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5159.651186] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5161.621434] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 11:56:45 (1787846205) [ 5168.448447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5168.707346] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5168.744507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5168.959130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5168.963359] Lustre: Skipped 6 previous similar messages [ 5174.833369] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5184.639627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5184.752264] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5184.792762] Lustre: 157467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5190.411259] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5200.520536] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5207.399806] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5215.180816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5215.468615] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5223.379507] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5225.177665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5232.084836] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5237.423562] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5237.782566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5242.850732] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5243.093574] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5245.928512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5245.947990] Lustre: Skipped 4 previous similar messages [ 5245.965424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5245.968587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5245.971048] LustreError: Skipped 3 previous similar messages [ 5245.994197] Lustre: Skipped 8 previous similar messages [ 5251.568617] Lustre: server umount lustre-OST0000 complete [ 5275.850498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5283.375615] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5292.737173] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5300.601635] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5300.841767] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5300.854705] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5300.856234] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.849288] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5307.978067] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5308.176622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5315.064329] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8e990dd8f800.ost_server_uuid 50 [ 5317.560708] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8e990dd8f800.ost_server_uuid in IDLE state after 0 sec [ 5318.221821] LustreError: 157473:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.60@tcp) returned error from blocking AST (req@000000006b46ea29 x1874691625337472 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 000000000510b1ee/0xba3a0b541acfc56 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.60@tcp remote: 0x1b49cf4937f21396 expref: 5 pid: 158478 timeout: 5418 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5318.256741] LustreError: lustre-MDT0000: A client on nid 192.168.201.60@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5318.266493] LustreError: 157463:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.60@tcp ns: mdt-lustre-MDT0000_UUID lock: 000000000510b1ee/0xba3a0b541acfc56 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.60@tcp remote: 0x1b49cf4937f21396 expref: 6 pid: 158478 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5330.669080] Lustre: server umount lustre-OST0000 complete [ 5341.736971] Lustre: server umount lustre-MDT0000 complete [ 5342.177433] LustreError: 160402:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5342.216380] LustreError: 160402:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 11 previous similar messages [ 5347.091517] LustreError: 157458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846391 with bad export cookie 838690655972423579 [ 5347.106572] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5347.127574] LustreError: 157458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5353.715907] Lustre: server umount lustre-MDT0001 complete [ 5366.413372] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 5370.224909] Key type lgssc unregistered [ 5370.615289] LNet: 163595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5370.626775] LNetError: 163595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5370.673821] LNet: Removed LNI 192.168.201.160@tcp [ 5371.995263] Key type .llcrypt unregistered [ 5371.998836] Key type ._llcrypt unregistered [ 5404.463322] Key type ._llcrypt registered [ 5404.467608] Key type .llcrypt registered [ 5404.628670] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 5406.390511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5406.615255] alg: No test for adler32 (adler32-zlib) [ 5407.942731] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 5408.326508] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 5410.049197] Key type lgssc registered [ 5412.774301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5422.759702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5422.792698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5424.469197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5430.538800] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5434.851217] LustreError: 165025:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5434.875676] LustreError: 165025:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5439.974710] LustreError: 165024:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5440.359923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5440.791584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5446.425123] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5455.066765] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5461.953544] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5469.691122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5470.255735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5479.224361] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5488.713468] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5494.850613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5505.244475] Lustre: Failing over lustre-MDT0000 [ 5505.621523] Lustre: server umount lustre-MDT0000 complete [ 5507.553891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5507.565164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5507.600916] LustreError: 165025:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5510.116338] LustreError: 165024:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5510.132364] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5515.250887] LustreError: 165019:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5515.298330] LustreError: 165019:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 4 previous similar messages [ 5518.339650] LustreError: 165007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.60@tcp arrived at 1787846563 with bad export cookie 14968760673515441819 [ 5525.476693] Lustre: 164326:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846554/real 1787846554] req@0000000039048812 x1874692882332544/t0(0) o400->MGC192.168.201.160@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787846570 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5525.497777] LustreError: 166175:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5525.520222] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5525.587813] LustreError: 166175:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 5 previous similar messages [ 5526.701608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5536.176195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5536.246076] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5541.365418] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5541.394807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5542.348872] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5544.991243] Lustre: lustre-MDT0000: Denying connection for new client cf45be1a-9ecb-4500-ac6e-5151461c2589 (at 192.168.201.60@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 5556.715097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5556.718307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5556.744838] Lustre: Skipped 2 previous similar messages [ 5556.759313] Lustre: Skipped 1 previous similar message [ 5559.800839] Lustre: server umount lustre-OST0000 complete [ 5564.348712] LustreError: 168778:0:(ldlm_lib.c:3067:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5564.359945] Lustre: 168127:0:(ldlm_lib.c:2467:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5564.367928] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5564.391585] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5564.403285] LustreError: 168127:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000ab684ce3 x1874692882353280/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5564.416190] LustreError: 168127:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5564.425930] LustreError: 168127:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5564.442662] Lustre: lustre-MDT0000: Recovery over after 0:23, of 2 clients 0 recovered and 2 were evicted. [ 5564.777367] Lustre: server umount lustre-MDT0000 complete [ 5566.956657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5566.982286] LustreError: 165019:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5567.022630] LustreError: 165019:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 6 previous similar messages [ 5568.968932] LustreError: 165006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846613 with bad export cookie 14968760673515442141 [ 5568.977619] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5568.978643] LustreError: 165006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5568.990267] LustreError: 164323:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5571.574452] Lustre: server umount lustre-MDT0001 complete [ 5586.247023] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 5590.061370] Key type lgssc unregistered [ 5590.474280] LNet: 169534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5590.491124] LNetError: 169534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5590.509981] LNet: Removed LNI 192.168.201.160@tcp [ 5591.918900] Key type .llcrypt unregistered [ 5591.924200] Key type ._llcrypt unregistered [ 5620.042451] Key type ._llcrypt registered [ 5620.043498] Key type .llcrypt registered [ 5620.112792] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 5620.970766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5620.994257] alg: No test for adler32 (adler32-zlib) [ 5622.028852] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 5622.210730] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 5623.824208] Key type lgssc registered [ 5625.633738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5635.012426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5635.024911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5636.534629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5641.241370] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5646.825448] LustreError: 170950:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5646.858609] LustreError: 170950:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5651.939718] LustreError: 170949:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5652.633053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5652.893986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5658.017625] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5669.714029] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5678.626695] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5687.256136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5687.812636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5696.935878] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5706.320600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5707.618588] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5716.453510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5716.455083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5716.487732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5716.516037] Lustre: Skipped 1 previous similar message [ 5720.788899] Lustre: server umount lustre-OST0000 complete [ 5726.693902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5726.709569] Lustre: Skipped 1 previous similar message [ 5726.726006] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5726.740310] Lustre: Skipped 1 previous similar message [ 5731.810988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5731.819993] Lustre: Skipped 1 previous similar message [ 5731.974755] Lustre: server umount lustre-MDT0000 complete [ 5736.874289] LustreError: 173731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846781 with bad export cookie 12083772717813295817 [ 5736.877170] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5736.888075] LustreError: 173731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5737.546875] Lustre: server umount lustre-MDT0001 complete [ 5751.833438] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 5755.404812] Key type lgssc unregistered [ 5755.881797] LNet: 174488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5755.898152] LNetError: 174488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5755.932941] LNet: Removed LNI 192.168.201.160@tcp [ 5757.042165] Key type .llcrypt unregistered [ 5757.046719] Key type ._llcrypt unregistered [ 5798.636564] Key type ._llcrypt registered [ 5798.640362] Key type .llcrypt registered [ 5798.785913] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 5800.526740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5800.635269] alg: No test for adler32 (adler32-zlib) [ 5801.755388] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 5802.067344] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 5803.753902] Key type lgssc registered [ 5805.030755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5814.835765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5814.855804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5816.624815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5822.794549] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5827.050893] LustreError: 176193:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5827.069668] LustreError: 176193:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 5832.173319] LustreError: 176194:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5833.296045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5833.963387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5839.352843] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5848.923555] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5855.608674] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5863.185118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5863.632547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5870.391865] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 5878.963507] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5887.218653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5897.713714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5897.716456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5897.736981] Lustre: Skipped 1 previous similar message [ 5897.765054] Lustre: Skipped 1 previous similar message [ 5902.377751] Lustre: server umount lustre-OST0000 complete [ 5907.941489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5907.953199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5907.960103] Lustre: Skipped 1 previous similar message [ 5913.057920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5913.070701] Lustre: Skipped 1 previous similar message [ 5918.178943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5918.184034] Lustre: Skipped 1 previous similar message [ 5921.248195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5921.502511] Lustre: server umount lustre-MDT0000 complete [ 5923.298728] LustreError: 178034:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5923.333855] LustreError: 178034:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 5925.843903] LustreError: 178695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846970 with bad export cookie 15566722557213438136 [ 5925.845132] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5925.851179] LustreError: 178695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5926.209456] Lustre: server umount lustre-MDT0001 complete [ 5939.703976] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 5942.733883] Key type lgssc unregistered [ 5943.049805] LNet: 179754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5943.062178] LNetError: 179754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5944.104744] LNet: Removed LNI 192.168.201.160@tcp [ 5945.177224] Key type .llcrypt unregistered [ 5945.182966] Key type ._llcrypt unregistered [ 5968.826550] Key type ._llcrypt registered [ 5968.830665] Key type .llcrypt registered [ 5968.946847] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 5969.996089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5970.121763] alg: No test for adler32 (adler32-zlib) [ 5971.249182] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 5971.440548] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 5973.080187] Key type lgssc registered [ 5974.314213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5985.704245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5985.728544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5987.313498] Lustre: *** cfs_fail_loc=135, val=0*** [ 5987.462219] LustreError: 181282:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5987.469549] LustreError: 181282:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.160@tcp: cfg command failed: rc = -2 [ 5987.484724] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5987.484724] [ 5987.498984] LustreError: MGC192.168.201.160@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5987.517225] LustreError: 181243:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5987.528246] LustreError: 181243:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5987.540305] LustreError: 181243:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5987.548340] LustreError: 181243:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5987.661185] Lustre: server umount lustre-MDT0000 complete [ 5987.662724] LustreError: 181243:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5993.557578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5994.040267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5998.377667] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6004.194591] LustreError: 181654:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6004.222248] LustreError: 181654:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6007.550792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6007.816706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6013.349881] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6024.032879] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6031.049394] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6033.379051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6033.382689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6033.388886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6037.475466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6037.488907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6039.289578] Lustre: server umount lustre-MDT0000 complete [ 6042.596241] LustreError: 181649:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6042.623815] LustreError: 181649:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6043.563729] LustreError: 181634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847088 with bad export cookie 9507224643032481489 [ 6043.579823] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6044.384066] Lustre: server umount lustre-MDT0001 complete [ 6060.401466] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 6064.328433] Key type lgssc unregistered [ 6064.660779] LNet: 183978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6064.668885] LNetError: 183978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6064.683700] LNet: Removed LNI 192.168.201.160@tcp [ 6065.908614] Key type .llcrypt unregistered [ 6065.914785] Key type ._llcrypt unregistered [ 6088.739247] Key type ._llcrypt registered [ 6088.745390] Key type .llcrypt registered [ 6088.895653] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 6089.355509] Key type .llcrypt unregistered [ 6089.357496] Key type ._llcrypt unregistered [ 6109.373191] Key type ._llcrypt registered [ 6109.375370] Key type .llcrypt registered [ 6109.573926] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 6110.993017] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6111.114224] alg: No test for adler32 (adler32-zlib) [ 6112.135678] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 6112.355593] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 6113.992189] Key type lgssc registered [ 6115.573830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6124.145252] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6124.161406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6139.680407] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6144.800347] LustreError: 186116:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.160@tcp: failed processing log, type 4: rc = -110 [ 6171.488212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6177.881742] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6185.654781] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6192.637377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6193.078591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6193.150518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6197.788957] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6198.244398] LustreError: 186142:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6198.260667] LustreError: 186142:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6203.364202] LustreError: 186141:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6205.426414] LustreError: 187151:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6205.457379] LustreError: 187151:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6205.860795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6206.273398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6210.681854] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6219.155864] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6228.201672] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6233.183740] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6247.395157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6247.400178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6247.414082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6247.435875] Lustre: Skipped 1 previous similar message [ 6252.515315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6252.529729] Lustre: Skipped 1 previous similar message [ 6253.443066] Lustre: server umount lustre-OST0000 complete [ 6257.633446] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6257.646518] Lustre: Skipped 1 previous similar message [ 6257.674106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6262.770821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6262.775915] Lustre: Skipped 1 previous similar message [ 6263.537927] Lustre: server umount lustre-MDT0000 complete [ 6267.090210] LustreError: 186122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847311 with bad export cookie 9101789367042740270 [ 6267.092593] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6267.107944] LustreError: 186122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6267.439661] Lustre: server umount lustre-MDT0001 complete [ 6278.614282] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 6281.619595] Key type lgssc unregistered [ 6281.997906] LNet: 190131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6282.012625] LNetError: 190131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6282.043186] LNet: Removed LNI 192.168.201.160@tcp [ 6283.253654] Key type .llcrypt unregistered [ 6283.255397] Key type ._llcrypt unregistered [ 6308.999721] Key type ._llcrypt registered [ 6309.006479] Key type .llcrypt registered [ 6309.106606] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 6309.988479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6310.176185] alg: No test for adler32 (adler32-zlib) [ 6311.362251] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 6311.740534] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 6313.584645] Key type lgssc registered [ 6314.723055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6321.732639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6321.741532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6323.148854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6327.551698] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6332.415187] LustreError: 191545:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6332.443884] LustreError: 191545:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6335.349654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6335.756743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6339.508292] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6345.840190] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6351.555875] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6358.317320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6358.646856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6365.820954] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6367.866362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6374.354800] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6380.275635] Lustre: Failing over lustre-MDT0000 [ 6380.529625] Lustre: server umount lustre-MDT0000 complete [ 6382.062227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6382.069091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6382.078982] LustreError: 191545:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6383.587717] LustreError: 191546:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6383.607225] LustreError: 191546:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 6383.620972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6387.622341] LustreError: 193822:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6387.638407] LustreError: 193822:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6392.746953] LustreError: 191542:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.60@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6392.777160] LustreError: 191542:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 6399.319329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6399.417940] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6399.562049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6399.603232] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6402.982838] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6403.401603] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6404.596977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6404.615796] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 6404.686799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6407.290262] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6414.695853] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6422.498396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6422.511243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6422.529049] Lustre: Skipped 1 previous similar message [ 6422.560245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6423.182539] Lustre: server umount lustre-OST0000 complete [ 6430.177954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6430.178699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6430.192733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6430.198684] Lustre: Skipped 1 previous similar message [ 6435.300282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6435.307435] Lustre: Skipped 2 previous similar messages [ 6436.380658] Lustre: server umount lustre-MDT0000 complete [ 6439.398338] LustreError: 191541:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6439.419766] LustreError: 191541:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 9 previous similar messages [ 6440.494453] LustreError: 191525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847485 with bad export cookie 14648866953966478394 [ 6440.506483] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6440.512342] LustreError: 191525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6440.931786] Lustre: server umount lustre-MDT0001 complete [ 6453.631359] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 6457.002123] Key type lgssc unregistered [ 6457.335630] LNet: 196432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6457.358218] LNetError: 196432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6457.372798] LNet: Removed LNI 192.168.201.160@tcp [ 6458.287232] Key type .llcrypt unregistered [ 6458.288978] Key type ._llcrypt unregistered [ 6481.907716] Key type ._llcrypt registered [ 6481.909188] Key type .llcrypt registered [ 6482.070299] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 6483.273544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6483.445360] alg: No test for adler32 (adler32-zlib) [ 6484.496645] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 6484.763986] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 6486.451618] Key type lgssc registered [ 6487.540075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6494.550262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6494.573664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6495.977045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6499.925213] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6505.447828] LustreError: 197839:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6505.461083] LustreError: 197839:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6508.316914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6508.677644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6513.060979] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6519.864996] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6525.054857] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6530.871557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6531.255742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6536.584905] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6540.475220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6543.283228] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6547.246904] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6554.686458] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6569.897600] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6579.681996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6579.696996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6579.733076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6581.738124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6581.747244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6581.765854] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6583.059929] Lustre: server umount lustre-OST0000 complete [ 6590.433609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6590.442134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6590.458268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6590.460856] Lustre: Skipped 1 previous similar message [ 6593.183663] Lustre: server umount lustre-MDT0000 complete [ 6596.636426] LustreError: 200369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847641 with bad export cookie 12138765324812781325 [ 6596.642723] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6596.647900] LustreError: 200369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6597.106347] Lustre: server umount lustre-MDT0001 complete [ 6607.805822] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 6610.267838] Key type lgssc unregistered [ 6610.515519] LNet: 201692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6610.522700] LNetError: 201692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6610.538463] LNet: Removed LNI 192.168.201.160@tcp [ 6611.444975] Key type .llcrypt unregistered [ 6611.446623] Key type ._llcrypt unregistered [ 6630.181448] Key type ._llcrypt registered [ 6630.183906] Key type .llcrypt registered [ 6630.292147] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 6631.175505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6631.300649] alg: No test for adler32 (adler32-zlib) [ 6632.343814] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 6632.559860] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 6634.256157] Key type lgssc registered [ 6635.270784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6642.970093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6642.986918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6644.429369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6647.976137] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6654.948102] LustreError: 202969:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6654.972525] LustreError: 202969:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6657.225788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6657.641402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6661.878839] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6669.237889] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6674.829846] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6681.823148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6682.157640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6689.133735] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6689.357388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6697.311968] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6703.448371] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6712.616822] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6719.835764] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6727.847670] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6739.429290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6739.435124] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6739.465121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6740.449116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6740.450549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6740.453968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6745.495985] Lustre: server umount lustre-OST0000 complete [ 6750.690710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6750.698662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6750.702294] Lustre: Skipped 1 previous similar message [ 6750.722380] Lustre: Skipped 2 previous similar messages [ 6755.816447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6755.825710] Lustre: Skipped 1 previous similar message [ 6756.381620] Lustre: server umount lustre-MDT0000 complete [ 6760.475492] LustreError: 205647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847805 with bad export cookie 15005218932474120799 [ 6760.476592] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6760.483233] LustreError: 205647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6760.797751] Lustre: server umount lustre-MDT0001 complete [ 6773.328553] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 6776.567674] Key type lgssc unregistered [ 6776.825551] LNet: 207847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6776.831140] LNetError: 207847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6776.854037] LNet: Removed LNI 192.168.201.160@tcp [ 6777.738254] Key type .llcrypt unregistered [ 6777.739975] Key type ._llcrypt unregistered [ 6802.721729] Key type ._llcrypt registered [ 6802.724348] Key type .llcrypt registered [ 6802.848711] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 6804.390724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6804.431915] alg: No test for adler32 (adler32-zlib) [ 6805.602968] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 6805.914670] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 6807.560225] Key type lgssc registered [ 6808.855333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6816.312291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6816.331573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6818.033118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6822.393870] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6827.526661] LustreError: 209259:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6827.550163] LustreError: 209259:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6830.999453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6831.516719] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6836.463612] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6843.467061] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6849.839091] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6857.310373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6857.944212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6863.036631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6864.194294] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6872.065542] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6892.010672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6892.011347] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6892.024220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6892.051356] Lustre: Skipped 1 previous similar message [ 6895.014901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6895.022990] Lustre: Skipped 1 previous similar message [ 6895.772969] Lustre: server umount lustre-OST0000 complete [ 6900.196808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6900.234885] Lustre: Skipped 1 previous similar message [ 6900.240841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6905.314162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6905.332120] Lustre: Skipped 2 previous similar messages [ 6906.049937] Lustre: server umount lustre-MDT0000 complete [ 6910.069233] LustreError: 209241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847954 with bad export cookie 13666665771280371451 [ 6910.076393] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6910.081344] LustreError: 209241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6910.657286] Lustre: server umount lustre-MDT0001 complete [ 6923.821710] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 6927.173455] Key type lgssc unregistered [ 6927.334540] LNet: 212947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6927.338797] LNetError: 212947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6927.353446] LNet: Removed LNI 192.168.201.160@tcp [ 6928.420368] Key type .llcrypt unregistered [ 6928.422041] Key type ._llcrypt unregistered [ 6951.241324] Key type ._llcrypt registered [ 6951.247810] Key type .llcrypt registered [ 6951.334798] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 6952.647756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6952.676059] alg: No test for adler32 (adler32-zlib) [ 6953.791168] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 6954.130241] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 6955.880184] Key type lgssc registered [ 6956.967437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6965.543802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6965.563804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.995114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6971.864132] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6976.490308] LustreError: 214216:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6976.518861] LustreError: 214216:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 6981.562371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6981.601904] LustreError: 214215:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6982.102928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6986.887674] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 6994.939957] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7000.391259] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7006.051590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7006.522534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7011.651284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7013.402484] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7022.305386] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7038.436416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7038.450962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7038.469929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7042.533728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7042.534601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7042.553985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7042.612206] Lustre: Skipped 1 previous similar message [ 7044.019833] Lustre: server umount lustre-OST0000 complete [ 7052.772687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7052.775432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7052.808205] Lustre: Skipped 1 previous similar message [ 7052.821258] Lustre: Skipped 1 previous similar message [ 7055.428798] Lustre: server umount lustre-MDT0000 complete [ 7057.892760] LustreError: 215799:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7057.910821] LustreError: 215799:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7059.504875] LustreError: 214196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848104 with bad export cookie 8474443255538690297 [ 7059.511054] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7059.517495] LustreError: 214196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7060.028461] Lustre: server umount lustre-MDT0001 complete [ 7071.320543] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 7074.383397] Key type lgssc unregistered [ 7074.741128] LNet: 217902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7074.746403] LNetError: 217902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7074.767181] LNet: Removed LNI 192.168.201.160@tcp [ 7075.662637] Key type .llcrypt unregistered [ 7075.664965] Key type ._llcrypt unregistered [ 7099.260351] Key type ._llcrypt registered [ 7099.263477] Key type .llcrypt registered [ 7099.448996] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 7100.957222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7101.001534] alg: No test for adler32 (adler32-zlib) [ 7102.072289] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 7102.391062] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 7104.128596] Key type lgssc registered [ 7105.590968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7114.741269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7114.809044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7116.464417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7121.453405] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7127.011423] LustreError: 219171:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7127.032843] LustreError: 219171:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7130.485454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7130.913515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7135.636381] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7145.116884] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7152.077618] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7159.892713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7160.394802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7161.410951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7167.184800] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7175.237525] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7185.462364] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7185.528464] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7185.860624] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7193.296836] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7202.562475] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7204.650693] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7216.928944] Lustre: Permanently deactivating lustre-OST0001 [ 7216.949401] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7227.548939] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7235.162561] Lustre: Permanently reactivating lustre-OST0001 [ 7235.175537] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7235.183706] Lustre: Skipped 2 previous similar messages [ 7236.533258] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7236.540174] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7236.545196] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7236.552169] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7236.595686] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7246.817400] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7246.824096] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7246.827278] LustreError: Skipped 1 previous similar message [ 7246.829494] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7246.842069] Lustre: Skipped 2 previous similar messages [ 7246.874556] Lustre: Skipped 1 previous similar message [ 7250.425265] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7252.903427] Lustre: server umount lustre-OST0001 complete [ 7255.535163] LustreError: 222729:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7255.576482] LustreError: 222729:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7260.646579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7260.647556] LustreError: 222729:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7260.652103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7260.652112] Lustre: Skipped 1 previous similar message [ 7260.681077] Lustre: Skipped 1 previous similar message [ 7260.701215] LustreError: 222729:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7265.775179] LustreError: 220993:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7265.796138] LustreError: 220993:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 3 previous similar messages [ 7265.919368] Lustre: server umount lustre-OST0000 complete [ 7270.882268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7270.906961] Lustre: Skipped 1 previous similar message [ 7270.907887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7270.916815] Lustre: Skipped 2 previous similar messages [ 7276.002515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7276.011085] Lustre: Skipped 1 previous similar message [ 7285.219296] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7285.221287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7285.242221] Lustre: Skipped 3 previous similar messages [ 7285.542652] Lustre: server umount lustre-MDT0000 complete [ 7290.345581] LustreError: 219166:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7290.372733] LustreError: 219166:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7291.737648] LustreError: 222008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848336 with bad export cookie 8053936210227218264 [ 7291.748674] LustreError: 222008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7291.752945] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7292.292745] Lustre: server umount lustre-MDT0001 complete [ 7305.043530] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 7309.089970] Key type lgssc unregistered [ 7309.438820] LNet: 224271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7309.445298] LNetError: 224271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7309.492164] LNet: Removed LNI 192.168.201.160@tcp [ 7310.820807] Key type .llcrypt unregistered [ 7310.825147] Key type ._llcrypt unregistered [ 7361.122207] Key type ._llcrypt registered [ 7361.123613] Key type .llcrypt registered [ 7361.297753] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 7362.336356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7362.416879] alg: No test for adler32 (adler32-zlib) [ 7363.617162] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 7363.862626] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 7365.528249] Key type lgssc registered [ 7367.083383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7376.514946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7376.532199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7377.935383] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7377.985389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7378.154934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7384.009374] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7393.929449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7394.090090] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7394.110744] Lustre: 226478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7394.385942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7399.889382] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7408.048568] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7413.863893] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7421.576617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7421.927701] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7422.300134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7428.804872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7429.912676] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7437.718479] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7441.799427] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7447.478710] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7454.275950] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7458.702599] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7467.886540] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7490.311107] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7490.316460] Lustre: Skipped 2 previous similar messages [ 7529.599822] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7529.641247] Lustre: Skipped 5 previous similar messages [ 7596.028454] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7596.033563] Lustre: Skipped 7 previous similar messages [ 7608.293817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7608.295534] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7608.322431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7608.335346] Lustre: Skipped 1 previous similar message [ 7612.056951] Lustre: server umount lustre-OST0000 complete [ 7618.530445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7618.558975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7618.563958] Lustre: Skipped 2 previous similar messages [ 7618.583381] Lustre: Skipped 1 previous similar message [ 7623.651685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7623.662069] Lustre: Skipped 1 previous similar message [ 7628.772838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7628.785232] Lustre: Skipped 1 previous similar message [ 7631.328734] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7631.718041] Lustre: server umount lustre-MDT0000 complete [ 7636.111151] LustreError: 226471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848680 with bad export cookie 11824283212265220687 [ 7636.120095] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7636.137027] LustreError: 226471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7636.658994] Lustre: server umount lustre-MDT0001 complete [ 7650.657629] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 7654.999911] Key type lgssc unregistered [ 7655.476071] LNet: 231281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7655.487714] LNetError: 231281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7655.511909] LNet: Removed LNI 192.168.201.160@tcp [ 7656.577415] Key type .llcrypt unregistered [ 7656.582287] Key type ._llcrypt unregistered [ 7686.875938] Key type ._llcrypt registered [ 7686.878956] Key type .llcrypt registered [ 7687.046535] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 7688.888392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7688.921662] alg: No test for adler32 (adler32-zlib) [ 7690.252518] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 7690.644286] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 7692.413255] Key type lgssc registered [ 7694.097496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7703.046423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7703.066300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7704.436752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7709.006821] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7714.788330] LustreError: 232696:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7714.819795] LustreError: 232696:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 2 previous similar messages [ 7718.488271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7718.812748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7723.808673] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7731.784781] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7737.773068] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7745.906231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7746.402760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7749.496526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7753.562236] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_default_debug -1 all [ 7762.459835] Lustre: DEBUG MARKER: oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7780.341427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7780.348150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7780.368738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7784.421149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7784.446160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7784.451866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7785.727152] Lustre: server umount lustre-OST0000 complete [ 7794.658549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7794.673993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7794.676628] Lustre: Skipped 1 previous similar message [ 7799.571576] Lustre: server umount lustre-MDT0000 complete [ 7799.795731] LustreError: 232697:0:(ldlm_lib.c:1228:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7799.821476] LustreError: 232697:0:(ldlm_lib.c:1228:target_handle_connect()) Skipped 1 previous similar message [ 7804.212942] LustreError: 232677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848848 with bad export cookie 1911303226445717015 [ 7804.220920] LustreError: MGC192.168.201.160@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7804.554814] Lustre: server umount lustre-MDT0001 complete [ 7818.126924] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 7821.076233] Key type lgssc unregistered [ 7821.403918] LNet: 236392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7821.417414] LNetError: 236392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7822.438810] LNet: Removed LNI 192.168.201.160@tcp [ 7823.281180] Key type .llcrypt unregistered [ 7823.283419] Key type ._llcrypt unregistered [ 7848.546883] Key type ._llcrypt registered [ 7848.548949] Key type .llcrypt registered [ 7848.693731] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 7850.815955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7850.886889] alg: No test for adler32 (adler32-zlib) [ 7852.221869] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 7852.524879] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 7854.384208] Key type lgssc registered [ 7855.848399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8126.618906] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 8129.505413] Key type lgssc unregistered [ 8129.757778] LNet: 238420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8129.772708] LNetError: 238420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8129.792202] LNet: Removed LNI 192.168.201.160@tcp [ 8130.525624] Key type .llcrypt unregistered [ 8130.527379] Key type ._llcrypt unregistered [ 8155.743945] Key type ._llcrypt registered [ 8155.747666] Key type .llcrypt registered [ 8155.878439] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing set_hostid [ 8168.640791] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing load_modules_local [ 8169.542828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8169.578996] alg: No test for adler32 (adler32-zlib) [ 8170.554277] Lustre: Lustre: Build Version: 2.17.57_98_gaf12cc4 [ 8170.806785] LNet: Added LNI 192.168.201.160@tcp [8/256/0/180] [ 8172.440185] Key type lgssc registered [ 8174.214158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8208.810763] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7879 sec ========== 12:47:31 (1787849251) [ 8210.729863] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:47:34 (1787849254) === [ 8214.606632] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:47:37 (1787849257) === [ 8259.671420] Lustre: DEBUG MARKER: oleg160-server.virtnet: executing unload_modules_local [ 8262.670271] Key type lgssc unregistered [ 8262.956371] LNet: 243659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8262.964399] LNetError: 243659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8262.979815] LNet: Removed LNI 192.168.201.160@tcp [ 8263.869963] Key type .llcrypt unregistered [ 8263.871811] Key type ._llcrypt unregistered