[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 522195325 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003266] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007796] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011155] LSM: Security Framework initializing [ 0.012073] Yama: becoming mindful. [ 0.013042] SELinux: Initializing. [ 0.015060] *** VALIDATE selinux *** [ 0.023450] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027865] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029003] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031129] *** VALIDATE tmpfs *** [ 0.033487] *** VALIDATE proc *** [ 0.035207] *** VALIDATE cgroup *** [ 0.036012] *** VALIDATE cgroup2 *** [ 0.038109] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039172] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041050] Spectre V2 : User space: Vulnerable [ 0.042010] Speculative Store Bypass: Vulnerable [ 0.045369] debug: unmapping init [mem 0xffffffffb7e59000-0xffffffffb7e60fff] [ 0.048180] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049720] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050026] ... version: 2 [ 0.051017] ... bit width: 48 [ 0.052014] ... generic registers: 4 [ 0.053016] ... value mask: 0000ffffffffffff [ 0.054020] ... max period: 00007fffffffffff [ 0.055015] ... fixed-purpose events: 3 [ 0.056028] ... event mask: 000000070000000f [ 0.057317] rcu: Hierarchical SRCU implementation. [ 0.059546] smp: Bringing up secondary CPUs ... [ 0.060608] x86: Booting SMP configuration: [ 0.061030] .... node #0, CPUs: #1 #2 #3 [ 0.064446] smp: Brought up 1 node, 4 CPUs [ 0.066014] smpboot: Max logical packages: 1 [ 0.067017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.123089] node 0 deferred pages initialised in 54ms [ 0.126104] devtmpfs: initialized [ 0.127208] x86/mm: Memory block size: 128MB [ 0.130335] gcov: version magic: 0x41383552 [ 0.133285] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.134074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.135276] pinctrl core: initialized pinctrl subsystem [ 0.136261] [ 0.136940] ************************************************************* [ 0.139017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.141012] ** ** [ 0.143013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.145013] ** ** [ 0.147014] ** This means that this kernel is built to expose internal ** [ 0.149015] ** IOMMU data structures, which may compromise security on ** [ 0.151012] ** your system. ** [ 0.153018] ** ** [ 0.156013] ** If you see this message and you are not debugging the ** [ 0.158013] ** kernel, report this immediately to your vendor! ** [ 0.160015] ** ** [ 0.162014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164018] ************************************************************* [ 0.166576] NET: Registered protocol family 16 [ 0.168391] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.171057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.176447] cpuidle: using governor menu [ 0.177499] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178339] PCI: Using configuration type 1 for base access [ 0.179104] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186192] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.188052] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.191183] cryptd: max_cpu_qlen set to 1000 [ 0.192249] ACPI: Added _OSI(Module Device) [ 0.193012] ACPI: Added _OSI(Processor Device) [ 0.194072] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.199577] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.206198] ACPI: Interpreter enabled [ 0.207068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.208017] ACPI: Using IOAPIC for interrupt routing [ 0.210138] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.213435] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223494] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.225061] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.228018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.231085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.236370] acpiphp: Slot [2] registered [ 0.238140] acpiphp: Slot [5] registered [ 0.239125] acpiphp: Slot [6] registered [ 0.241128] acpiphp: Slot [7] registered [ 0.243193] acpiphp: Slot [8] registered [ 0.245179] acpiphp: Slot [9] registered [ 0.247107] acpiphp: Slot [10] registered [ 0.249152] acpiphp: Slot [3] registered [ 0.251123] acpiphp: Slot [4] registered [ 0.253107] acpiphp: Slot [11] registered [ 0.255122] acpiphp: Slot [12] registered [ 0.256148] acpiphp: Slot [13] registered [ 0.259129] acpiphp: Slot [14] registered [ 0.260128] acpiphp: Slot [15] registered [ 0.262116] acpiphp: Slot [16] registered [ 0.264128] acpiphp: Slot [17] registered [ 0.266130] acpiphp: Slot [18] registered [ 0.268121] acpiphp: Slot [19] registered [ 0.270120] acpiphp: Slot [20] registered [ 0.272132] acpiphp: Slot [21] registered [ 0.274073] acpiphp: Slot [22] registered [ 0.276101] acpiphp: Slot [23] registered [ 0.277088] acpiphp: Slot [24] registered [ 0.278060] acpiphp: Slot [25] registered [ 0.279037] acpiphp: Slot [26] registered [ 0.280105] acpiphp: Slot [27] registered [ 0.282097] acpiphp: Slot [28] registered [ 0.283124] acpiphp: Slot [29] registered [ 0.284094] acpiphp: Slot [30] registered [ 0.286089] acpiphp: Slot [31] registered [ 0.287055] PCI host bridge to bus 0000:00 [ 0.289021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.291022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.293026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.296023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.298020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.301025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.302194] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.303880] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.307503] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.320860] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.325056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.328021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.330016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.336435] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.340120] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.343075] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.345842] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.351016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.365015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.370015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.375741] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.399018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.413017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.439020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.448272] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.460017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.472017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.495018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.503205] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.511019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.520020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.531017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.537820] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.541013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.546011] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.559015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.571070] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.575013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.582017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.594015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.601110] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.605012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.611017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.623013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.631265] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.633238] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.634260] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.636201] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.637128] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.641196] iommu: Default domain type: Passthrough [ 0.642459] SCSI subsystem initialized [ 0.643157] ACPI: bus type USB registered [ 0.644145] usbcore: registered new interface driver usbfs [ 0.646080] usbcore: registered new interface driver hub [ 0.648078] usbcore: registered new device driver usb [ 0.649225] pps_core: LinuxPPS API ver. 1 registered [ 0.651011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.654053] PTP clock support registered [ 0.656037] EDAC MC: Ver: 3.0.0 [ 0.657487] PCI: Using ACPI for IRQ routing [ 0.659800] NetLabel: Initializing [ 0.661021] NetLabel: domain hash size = 128 [ 0.662010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.664080] NetLabel: unlabeled traffic allowed by default [ 0.666136] vgaarb: loaded [ 0.667243] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.669014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.676818] clocksource: Switched to clocksource kvm-clock [ 0.778087] VFS: Disk quotas dquot_6.6.0 [ 0.779525] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.782181] *** VALIDATE ramfs *** [ 0.783309] *** VALIDATE hugetlbfs *** [ 0.784530] pnp: PnP ACPI init [ 0.786552] pnp: PnP ACPI: found 6 devices [ 0.807352] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.810506] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.812794] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.814947] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.817542] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.819732] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.822457] NET: Registered protocol family 2 [ 0.824667] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.828491] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.831729] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.836622] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.840052] TCP: Hash tables configured (established 65536 bind 65536) [ 0.843042] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.846225] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.849136] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.852176] NET: Registered protocol family 1 [ 0.854851] RPC: Registered named UNIX socket transport module. [ 0.856788] RPC: Registered udp transport module. [ 0.858531] RPC: Registered tcp transport module. [ 0.860216] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.862506] NET: Registered protocol family 44 [ 0.864046] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.865864] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.867807] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.870763] PCI: CLS 0 bytes, default 64 [ 0.872923] Unpacking initramfs... [ 2.234886] debug: unmapping init [mem 0xffff99273cc54000-0xffff99273ffbffff] [ 2.238607] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.240493] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.242796] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.719840] Initialise system trusted keyrings [ 2.721605] Key type blacklist registered [ 2.723574] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.732454] zbud: loaded [ 2.735587] *** VALIDATE nfs *** [ 2.736727] *** VALIDATE nfs4 *** [ 2.738502] pstore: using deflate compression [ 2.742559] Platform Keyring initialized [ 2.835101] NET: Registered protocol family 38 [ 2.836436] Key type asymmetric registered [ 2.837370] Asymmetric key parser 'x509' registered [ 2.838831] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.841129] io scheduler mq-deadline registered [ 2.842459] io scheduler kyber registered [ 2.843839] io scheduler bfq registered [ 2.846181] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.848701] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.851189] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.853685] ACPI: Power Button [PWRF] [ 2.859546] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.865929] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.879694] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.886738] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.903206] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.930763] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.959037] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.963472] Non-volatile memory driver v1.3 [ 2.964819] Linux agpgart interface v0.103 [ 2.991260] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 2.993841] vda: detected capacity change from 0 to 74829824 [ 3.019879] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.022418] vdb: detected capacity change from 0 to 1073741824 [ 3.045025] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.048328] vdc: detected capacity change from 0 to 2621440000 [ 3.068162] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.070501] vdd: detected capacity change from 0 to 2621440000 [ 3.093319] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.095570] vde: detected capacity change from 0 to 4294967296 [ 3.122500] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.126700] vdf: detected capacity change from 0 to 4294967296 [ 3.158673] libphy: Fixed MDIO Bus: probed [ 3.208549] usbcore: registered new interface driver usbserial_generic [ 3.211610] usbserial: USB Serial support registered for generic [ 3.214527] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.219412] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.221479] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.224413] mousedev: PS/2 mouse device common for all mice [ 3.226773] rtc_cmos 00:05: RTC can wake from S4 [ 3.229392] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.235202] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.237082] rtc_cmos 00:05: registered as rtc0 [ 3.242944] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.247144] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.257723] intel_pstate: CPU model not supported [ 3.272933] hid: raw HID events driver (C) Jiri Kosina [ 3.274920] usbcore: registered new interface driver usbhid [ 3.286411] usbhid: USB HID core driver [ 3.292311] drop_monitor: Initializing network drop monitor service [ 3.302105] Initializing XFRM netlink socket [ 3.305649] NET: Registered protocol family 10 [ 3.319208] Segment Routing with IPv6 [ 3.320407] NET: Registered protocol family 17 [ 3.323735] mpls_gso: MPLS GSO support [ 3.330194] RAS: Correctable Errors collector initialized. [ 3.332218] AVX version of gcm_enc/dec engaged. [ 3.333833] AES CTR mode by8 optimization enabled [ 3.511589] sched_clock: Marking stable (3511422426, 0)->(4463983964, -952561538) [ 3.517992] registered taskstats version 1 [ 3.520888] Loading compiled-in X.509 certificates [ 3.525376] zswap: loaded using pool lzo/zbud [ 3.569793] Key type big_key registered [ 3.606951] Key type encrypted registered [ 3.618411] ima: No TPM chip found, activating TPM-bypass! [ 3.621161] ima: Allocated hash algorithm: sha1 [ 3.625116] ima: No architecture policies found [ 3.626944] evm: Initialising EVM extended attributes: [ 3.632227] evm: security.selinux [ 3.633696] evm: security.ima [ 3.635772] evm: security.capability [ 3.638422] evm: HMAC attrs: 0x1 [ 3.641671] rtc_cmos 00:05: setting system clock to 2026-08-24 23:21:24 UTC (1787613684) [ 3.653326] debug: unmapping init [mem 0xffffffffb8e03000-0xffffffffb8ffffff] [ 3.660199] debug: unmapping init [mem 0xffffffffb7b82000-0xffffffffb7e58fff] [ 3.694636] Write protecting the kernel read-only data: 28672k [ 3.702170] debug: unmapping init [mem 0xffffffffb6203000-0xffffffffb63fffff] [ 3.722533] debug: unmapping init [mem 0xffffffffb6b14000-0xffffffffb6bfffff] [ 3.853777] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.883556] systemd[1]: Detected virtualization kvm. [ 3.886120] systemd[1]: Detected architecture x86-64. [ 3.891469] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.966969] systemd[1]: No hostname configured. [ 3.973632] systemd[1]: Set hostname to . [ 3.975492] random: systemd: uninitialized urandom read (16 bytes read) [ 3.984639] systemd[1]: Initializing machine ID from random generator. [ 4.189404] random: ln: uninitialized urandom read (6 bytes read) [ 4.415150] random: systemd: uninitialized urandom read (16 bytes read) [ 4.418298] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.425765] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.440252] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.258652] device-mapper: uevent: version 1.0.3 [ 6.263567] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ 7.848693] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.312040] virtio_net virtio0 ens2: renamed from eth0 [ 8.585294] scsi host0: ata_piix [ 8.604849] scsi host1: ata_piix [ 8.606189] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.618111] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.154306] random: crng init done [ 14.156438] random: 7 urandom warning(s) missed due to ratelimiting [ 16.260984] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.079592] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.317551] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.135227] SELinux: Disabled at runtime. [ 22.286948] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.328830] systemd[1]: Detected virtualization kvm. [ 22.333122] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.217120] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.230946] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.259728] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.269196] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.274554] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.300836] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.325190] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ 24.460905] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 26.117281] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 27.228930] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.273447] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 27.824877] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 28.026144] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit)[ 32.132182] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit)[ 32.885619] NFS: Registering the id_resolver key type [ 32.894164] Key type id_resolver registered [ 32.896423] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg156-server login: [ 97.183664] libcfs: loading out-of-tree module taints kernel. [ 97.221954] Key type ._llcrypt registered [ 97.224776] Key type .llcrypt registered [ 97.316935] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 117.194505] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 119.405646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 119.423110] alg: No test for adler32 (adler32-zlib) [ 120.980484] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 122.120214] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 124.057089] Key type lgssc registered [ 126.503915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 149.010759] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 152.374038] hrtimer: interrupt took 6842736 ns [ 199.238667] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 214.983324] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 215.061056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.473415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 216.510627] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 216.625882] Lustre: lustre-MDT0000: new disk, initializing [ 216.711467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 216.732750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 221.831788] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.666031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 236.813401] Lustre: 6522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 236.874553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 236.886083] Lustre: Skipped 1 previous similar message [ 237.028491] Lustre: lustre-MDT0001: new disk, initializing [ 237.150470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 237.205874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 237.225826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 243.853835] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.167572] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 260.647908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 261.190368] Lustre: lustre-OST0000: new disk, initializing [ 261.193878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 261.200797] Lustre: 8460:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 261.309738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 268.496428] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 269.856959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 269.861641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 269.951769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 285.567262] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 285.745786] Lustre: lustre-OST0001: new disk, initializing [ 285.761776] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 285.781469] Lustre: 9534:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 285.874496] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 292.431080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 292.447147] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 292.545548] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 293.361787] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 306.771778] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 316.365171] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 324.101866] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing check_logdir /tmp/testlogs/ [ 329.453171] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing yml_node [ 334.343494] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 337.381955] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 340.064884] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 342.097692] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 19:27:01 EDT 2026 [ 365.742942] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 367.765630] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 384.997379] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 385.004213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 385.018575] Lustre: Skipped 2 previous similar messages [ 385.019926] Lustre: Skipped 1 previous similar message [ 390.114451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 390.130253] Lustre: Skipped 5 previous similar messages [ 390.843152] Lustre: server umount lustre-MDT0000 complete [ 395.355107] LustreError: 6513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614076 with bad export cookie 3190104875076051190 [ 395.359261] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 395.360846] LustreError: 6513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 400.353139] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 400.359767] LustreError: 12280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 400.379847] Lustre: Skipped 1 previous similar message [ 400.394813] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 400.403281] LustreError: 12280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 400.418188] Lustre: Skipped 1 previous similar message [ 401.858983] Lustre: server umount lustre-MDT0001 complete [ 406.192610] Lustre: server umount lustre-OST0000 complete [ 410.236548] Lustre: server umount lustre-OST0001 complete [ 417.729900] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 428.852357] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 467.680625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 467.981287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 468.018359] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 468.121543] Lustre: lustre-MDT0000: new disk, initializing [ 468.228680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.242868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 473.534991] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 485.975754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.230716] Lustre: 15090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 486.332635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 486.337530] Lustre: Skipped 1 previous similar message [ 486.450367] Lustre: lustre-MDT0001: new disk, initializing [ 486.541043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 486.587990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 486.605495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 492.629123] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 503.269191] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 509.918960] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 518.585692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 518.935612] Lustre: lustre-OST0000: new disk, initializing [ 518.938539] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 518.947244] Lustre: 17059:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 519.086826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 520.578425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 520.583907] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 520.745093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 526.621517] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 538.530571] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 543.815460] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 543.961682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 549.742065] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 549.942141] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 556.515383] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.522419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 556.537884] Lustre: Skipped 1 previous similar message [ 556.544368] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 556.552480] Lustre: Skipped 1 previous similar message [ 558.501743] Lustre: server umount lustre-OST0000 complete [ 563.681759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 563.691680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.715109] Lustre: Skipped 1 previous similar message [ 563.725142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 569.721444] Lustre: server umount lustre-MDT0000 complete [ 571.877672] LustreError: 15102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.904380] LustreError: 15102:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 573.837220] LustreError: 15082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614254 with bad export cookie 3190104875076052723 [ 573.840492] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.846203] LustreError: 15082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 574.268341] Lustre: server umount lustre-MDT0001 complete [ 578.184989] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 19:30:57 (1787614257) [ 585.652743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 586.046744] LustreError: 19325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 586.140877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 590.724496] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 591.332306] LustreError: 19326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.429177] LustreError: 19325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.393837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 605.304324] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 613.623653] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 619.985407] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 628.201148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 635.574929] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 645.025023] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 655.333251] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 655.339425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 655.346750] Lustre: Skipped 2 previous similar messages [ 655.346790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 655.354585] Lustre: Skipped 2 previous similar messages [ 660.414657] Lustre: server umount lustre-OST0000 complete [ 666.595150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 666.605148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 672.088870] Lustre: server umount lustre-MDT0000 complete [ 675.811113] LustreError: 19326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 675.862820] LustreError: 19326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 677.001843] LustreError: 19305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614357 with bad export cookie 3190104875076053605 [ 677.005205] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 677.018925] LustreError: 19305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 677.541049] Lustre: server umount lustre-MDT0001 complete [ 692.051418] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 696.340781] Key type lgssc unregistered [ 696.804594] LNet: 22867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.811934] LNetError: 22867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.831913] LNet: Removed LNI 192.168.201.156@tcp [ 698.002127] Key type .llcrypt unregistered [ 698.005125] Key type ._llcrypt unregistered [ 729.540656] Key type ._llcrypt registered [ 729.549118] Key type .llcrypt registered [ 729.783807] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 731.008745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 731.032100] alg: No test for adler32 (adler32-zlib) [ 732.292402] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 732.697244] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 734.385090] Key type lgssc registered [ 735.790878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 745.271635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 745.309483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 746.679010] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 750.470650] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 756.199838] LustreError: 24282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 756.232687] LustreError: 24282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 758.949345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 759.359849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 764.818531] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 775.486324] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 782.708619] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 791.240842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 791.670549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 792.708667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 798.827818] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 808.030893] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 825.831081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 825.834108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 825.838208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 825.862727] Lustre: Skipped 1 previous similar message [ 830.958364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 830.961252] Lustre: Skipped 1 previous similar message [ 831.979572] Lustre: server umount lustre-OST0000 complete [ 836.067139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 836.073327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 836.077364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 836.077374] Lustre: Skipped 1 previous similar message [ 836.122873] Lustre: Skipped 1 previous similar message [ 841.190444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 841.202170] Lustre: Skipped 2 previous similar messages [ 846.308092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 850.402118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 850.694631] Lustre: server umount lustre-MDT0000 complete [ 851.425763] LustreError: 25011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 851.449676] LustreError: 25011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 854.702390] LustreError: 24263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614535 with bad export cookie 2871993720855481706 [ 854.702993] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 854.720864] LustreError: 24263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 855.120623] Lustre: server umount lustre-MDT0001 complete [ 868.835816] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 872.415839] Key type lgssc unregistered [ 872.759645] LNet: 28264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.767781] LNetError: 28264:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.787898] LNet: Removed LNI 192.168.201.156@tcp [ 873.571572] Key type .llcrypt unregistered [ 873.574677] Key type ._llcrypt unregistered [ 897.552626] Key type ._llcrypt registered [ 897.556749] Key type .llcrypt registered [ 897.668970] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 898.809853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 898.934525] alg: No test for adler32 (adler32-zlib) [ 900.096924] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 900.322270] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 901.960172] Key type lgssc registered [ 903.146256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 909.726770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 909.747957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 911.218479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 915.348427] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 921.574738] LustreError: 29680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 921.595875] LustreError: 29680:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 924.430589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 924.861236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 929.747724] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 937.604472] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 944.911892] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 960.553715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 961.078810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 968.200356] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 977.474760] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 981.560337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 989.671789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 989.678867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 989.687510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 991.720527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.722581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 991.755293] Lustre: Skipped 1 previous similar message [ 995.426123] Lustre: server umount lustre-OST0000 complete [ 1001.441724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1001.446919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.482830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1006.084559] Lustre: server umount lustre-MDT0000 complete [ 1007.076480] LustreError: 29680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1007.102085] LustreError: 29680:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1010.329180] LustreError: 29660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614691 with bad export cookie 13899340102747092470 [ 1010.340364] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1010.343414] LustreError: 29660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1010.725665] Lustre: server umount lustre-MDT0001 complete [ 1023.326377] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1026.623251] Key type lgssc unregistered [ 1026.986783] LNet: 33565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.995752] LNetError: 33565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.022418] LNet: Removed LNI 192.168.201.156@tcp [ 1028.046375] Key type .llcrypt unregistered [ 1028.048974] Key type ._llcrypt unregistered [ 1059.509333] Key type ._llcrypt registered [ 1059.513575] Key type .llcrypt registered [ 1059.632339] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1060.886425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1060.982893] alg: No test for adler32 (adler32-zlib) [ 1062.215304] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 1062.506646] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1064.272229] Key type lgssc registered [ 1065.446291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1074.782755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1074.800899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1076.232354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1080.000110] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1086.434571] LustreError: 34987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1086.459904] LustreError: 34987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1089.571192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.845420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1094.148485] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1102.665216] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1110.070449] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1118.080684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1118.658264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.713037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1125.951806] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1134.407050] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1145.315032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1145.321535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1145.336131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1145.342946] Lustre: Skipped 1 previous similar message [ 1145.360333] Lustre: Skipped 1 previous similar message [ 1149.616080] Lustre: server umount lustre-OST0000 complete [ 1155.557184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1155.577803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1155.594049] Lustre: Skipped 1 previous similar message [ 1160.674595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1161.635536] Lustre: server umount lustre-MDT0000 complete [ 1165.811563] LustreError: 36013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1165.828218] LustreError: 36013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1167.106130] LustreError: 34969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787614847 with bad export cookie 1942550909922183161 [ 1167.129472] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1167.860750] Lustre: server umount lustre-MDT0001 complete [ 1184.538815] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1187.055671] Key type lgssc unregistered [ 1187.450654] LNet: 38525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1187.462443] LNetError: 38525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1188.520970] LNet: Removed LNI 192.168.201.156@tcp [ 1189.162083] Key type .llcrypt unregistered [ 1189.164348] Key type ._llcrypt unregistered [ 1217.770902] Key type ._llcrypt registered [ 1217.775420] Key type .llcrypt registered [ 1217.886785] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1218.918204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1219.042648] alg: No test for adler32 (adler32-zlib) [ 1220.086524] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 1220.346628] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1222.048174] Key type lgssc registered [ 1223.517659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1232.692797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1232.726944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1234.486030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1240.046823] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1244.654736] LustreError: 39941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.681105] LustreError: 39941:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1249.763106] LustreError: 39941:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1250.399214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.832818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1255.974951] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1265.237581] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1272.262033] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1279.812997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1280.337309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.005404] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1295.519701] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1305.015796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1315.297602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1315.305560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1315.315040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1315.341535] Lustre: Skipped 1 previous similar message [ 1315.366569] Lustre: Skipped 1 previous similar message [ 1317.926249] Lustre: lustre-OST0000: Not available for connect from 192.168.201.56@tcp (stopping) [ 1319.552789] Lustre: server umount lustre-OST0000 complete [ 1340.900367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.906068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1340.917252] Lustre: Skipped 1 previous similar message [ 1346.018381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1346.028157] Lustre: Skipped 2 previous similar messages [ 1346.555805] Lustre: server umount lustre-MDT0000 complete [ 1350.089826] LustreError: 39921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787615030 with bad export cookie 8549476366669771645 [ 1350.097946] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1350.101048] LustreError: 39921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1356.835192] Lustre: server umount lustre-MDT0001 complete [ 1370.229295] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1373.555565] Key type lgssc unregistered [ 1373.854551] LNet: 43674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.870618] LNetError: 43674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1373.904462] LNet: Removed LNI 192.168.201.156@tcp [ 1374.682757] Key type .llcrypt unregistered [ 1374.684460] Key type ._llcrypt unregistered [ 1398.563539] Key type ._llcrypt registered [ 1398.565208] Key type .llcrypt registered [ 1398.666714] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1399.686321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1399.938027] alg: No test for adler32 (adler32-zlib) [ 1400.955898] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 1401.159408] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1402.848220] Key type lgssc registered [ 1404.774105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1413.936938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1413.951367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.505751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1420.027480] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1425.892064] LustreError: 45105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1425.915130] LustreError: 45105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1429.768279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1430.048351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1435.361665] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1444.554742] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1450.220532] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1457.191035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1457.711802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.797661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1464.588516] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1472.654960] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1480.184677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1480.210105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1480.217489] Lustre: Skipped 1 previous similar message [ 1480.611043] Lustre: server umount lustre-MDT0000 complete [ 1480.767594] LustreError: 45100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1484.930245] LustreError: 45085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787615165 with bad export cookie 13387072447278091909 [ 1484.947015] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1485.285212] LustreError: 45102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1485.289495] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1485.320037] Lustre: Skipped 2 previous similar messages [ 1485.325219] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1485.333941] Lustre: Skipped 1 previous similar message [ 1489.314729] LustreError: 45102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1489.318054] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1489.333339] LustreError: 45102:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1489.336726] Lustre: Skipped 1 previous similar message [ 1491.579864] Lustre: server umount lustre-MDT0001 complete [ 1513.257474] Lustre: server umount lustre-OST0000 complete [ 1530.898868] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1533.203880] Key type lgssc unregistered [ 1533.405594] LNet: 48969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1533.408642] LNetError: 48969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1533.426218] LNet: Removed LNI 192.168.201.156@tcp [ 1534.257898] Key type .llcrypt unregistered [ 1534.260559] Key type ._llcrypt unregistered [ 1564.084504] Key type ._llcrypt registered [ 1564.087108] Key type .llcrypt registered [ 1564.213195] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1565.379231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1565.478060] alg: No test for adler32 (adler32-zlib) [ 1566.511762] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 1566.873607] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1568.584194] Key type lgssc registered [ 1569.932386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1578.490338] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1578.510800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1594.080684] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1599.200377] LustreError: 50358:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 1625.888461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1632.805633] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1640.513240] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1806.613376] Lustre: server umount lustre-OST0000 complete [ 1821.585690] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1824.158740] Key type lgssc unregistered [ 1824.398965] LNet: 52034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1824.406279] LNetError: 52034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1824.442678] LNet: Removed LNI 192.168.201.156@tcp [ 1825.337846] Key type .llcrypt unregistered [ 1825.340566] Key type ._llcrypt unregistered [ 1848.160435] Key type ._llcrypt registered [ 1848.164036] Key type .llcrypt registered [ 1848.237989] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1849.240050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1849.416829] alg: No test for adler32 (adler32-zlib) [ 1850.534367] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 1850.876869] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 1852.649582] Key type lgssc registered [ 1854.915604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1863.562767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1863.583215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.074514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1869.928450] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1874.411325] LustreError: 53454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1874.458362] LustreError: 53454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1878.917328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.276932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1884.010112] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1892.260673] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1899.176545] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1907.405915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1907.681493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1909.787935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1914.072476] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 1921.382923] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1924.708503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1924.711609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1930.171710] Lustre: server umount lustre-OST0000 complete [ 1935.842255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1935.848495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1935.865024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1935.872111] Lustre: Skipped 1 previous similar message [ 1940.453280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1940.455529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1940.502838] Lustre: Skipped 1 previous similar message [ 1945.572617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.581940] Lustre: Skipped 1 previous similar message [ 1950.176156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1950.434936] Lustre: server umount lustre-MDT0000 complete [ 1950.695996] LustreError: 54697:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1950.727356] LustreError: 54697:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1954.323240] LustreError: 53434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787615635 with bad export cookie 12447545436919901255 [ 1954.335178] LustreError: 53434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1954.336837] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1954.952855] Lustre: server umount lustre-MDT0001 complete [ 1966.413820] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 1969.184138] Key type lgssc unregistered [ 1969.489888] LNet: 56995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1969.500948] LNetError: 56995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1970.534608] LNet: Removed LNI 192.168.201.156@tcp [ 1971.544856] Key type .llcrypt unregistered [ 1971.549322] Key type ._llcrypt unregistered [ 1997.455755] Key type ._llcrypt registered [ 1997.457752] Key type .llcrypt registered [ 1997.632968] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 1999.201343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1999.259614] alg: No test for adler32 (adler32-zlib) [ 2000.378613] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 2000.625620] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2002.299195] Key type lgssc registered [ 2003.738210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2013.059325] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2013.091589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2028.704879] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2033.824413] LustreError: 58381:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 2060.512492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2066.564555] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2073.503586] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2080.630837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.784584] LustreError: 57716:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000011eef3e0 x1874450900322560/t0(0) o250->MGC192.168.201.156@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2081.076988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.147814] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2085.377958] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2086.371033] LustreError: 58407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2086.388505] LustreError: 58407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2091.509013] LustreError: 58406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2093.905190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2094.141420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2098.356274] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2105.718913] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2112.095370] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2114.532471] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2114.542622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.553935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2116.580464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2116.584626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2121.728111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2121.736402] Lustre: Skipped 2 previous similar messages [ 2126.827599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2126.840799] Lustre: Skipped 1 previous similar message [ 2128.353992] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2128.611567] Lustre: server umount lustre-OST0000 complete [ 2152.419069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.423080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2152.441706] Lustre: Skipped 1 previous similar message [ 2152.461610] Lustre: Skipped 1 previous similar message [ 2157.776816] Lustre: server umount lustre-MDT0000 complete [ 2161.648126] LustreError: 58388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787615842 with bad export cookie 11355170783378713818 [ 2161.654138] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2161.664188] LustreError: 58388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2168.123732] Lustre: server umount lustre-MDT0001 complete [ 2179.534746] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2182.649335] Key type lgssc unregistered [ 2182.993628] LNet: 62132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2183.010716] LNetError: 62132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2183.037497] LNet: Removed LNI 192.168.201.156@tcp [ 2184.115578] Key type .llcrypt unregistered [ 2184.118140] Key type ._llcrypt unregistered [ 2207.589910] Key type ._llcrypt registered [ 2207.594523] Key type .llcrypt registered [ 2207.655137] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2208.825319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2208.844909] alg: No test for adler32 (adler32-zlib) [ 2209.839891] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 2210.000670] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2211.696168] Key type lgssc registered [ 2212.854720] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2220.461667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2220.481355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.758145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2225.698122] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2231.268120] LustreError: 63547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2231.283376] LustreError: 63547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2235.009512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2235.522736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2240.456546] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2249.025770] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2255.370213] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2262.768795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2263.124075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2267.251200] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2269.852963] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2278.947807] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2296.696461] Lustre: server umount lustre-OST0000 complete [ 2301.921085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2301.932949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2301.949655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2302.437560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2302.452769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2302.466619] Lustre: Skipped 1 previous similar message [ 2307.555501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2307.565978] Lustre: Skipped 1 previous similar message [ 2310.688569] Lustre: server umount lustre-MDT0000 complete [ 2312.681763] LustreError: 64788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2312.705285] LustreError: 64788:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2313.696156] Lustre: 62859:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787615978/real 1787615978] req@00000000a42ed840 x1874451119502848/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787615994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2313.696254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2313.730514] Lustre: 62859:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2313.766862] LustreError: 63542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2313.784241] LustreError: 63542:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2314.869363] LustreError: 66147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787615995 with bad export cookie 17379192688450068092 [ 2314.869885] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2314.876304] LustreError: 66147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2315.375341] Lustre: server umount lustre-MDT0001 complete [ 2329.129995] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2332.213614] Key type lgssc unregistered [ 2332.570614] LNet: 67105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2332.577583] LNetError: 67105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2332.603342] LNet: Removed LNI 192.168.201.156@tcp [ 2333.776888] Key type .llcrypt unregistered [ 2333.782549] Key type ._llcrypt unregistered [ 2361.901256] Key type ._llcrypt registered [ 2361.904656] Key type .llcrypt registered [ 2362.004933] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2363.193776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2363.210300] alg: No test for adler32 (adler32-zlib) [ 2364.276340] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 2364.516795] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2366.160220] Key type lgssc registered [ 2367.315184] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2376.928930] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 20:00:56 (1787616056) [ 2383.987439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2384.001587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2385.379642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2389.643643] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2395.619368] LustreError: 68824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.649935] LustreError: 68824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2397.955600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.248343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2402.947948] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2410.877772] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2417.798481] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2426.454910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2426.779900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2430.936697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2434.242086] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2443.196165] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2448.873045] Lustre: Failing over lustre-MDT0000 [ 2449.167682] Lustre: server umount lustre-MDT0000 complete [ 2449.394932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2449.410975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.434302] LustreError: 68824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.479910] LustreError: 68821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.474296] LustreError: 68825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2453.474541] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2458.606257] LustreError: 69826:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.634466] LustreError: 69826:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2460.000276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.217790] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2460.367575] Lustre: *** cfs_fail_loc=135, val=0*** [ 2460.511112] LustreError: 71911:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2460.524817] LustreError: 71911:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.156@tcp: cfg command failed: rc = -2 [ 2460.535610] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2460.535610] [ 2460.542631] LustreError: MGC192.168.201.156@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2460.553654] LustreError: 71899:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2460.562717] LustreError: 71899:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 2460.585743] LustreError: 71899:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2460.591613] LustreError: 71899:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2460.635704] Lustre: server umount lustre-MDT0000 complete [ 2460.648380] LustreError: 71899:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2466.830378] LustreError: 68821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2466.912270] LustreError: 68821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2468.957876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.125923] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2469.597719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2469.691847] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2470.888603] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2475.011775] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2475.049985] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2475.117669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2475.714257] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2484.193172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2484.200877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2484.225845] Lustre: Skipped 1 previous similar message [ 2484.237694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2485.246273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2488.993405] Lustre: server umount lustre-OST0000 complete [ 2495.458204] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2495.464971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2495.497434] Lustre: Skipped 1 previous similar message [ 2495.518503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2495.518513] Lustre: Skipped 1 previous similar message [ 2500.582122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2500.594076] Lustre: Skipped 2 previous similar messages [ 2501.286098] Lustre: server umount lustre-MDT0000 complete [ 2505.704817] LustreError: 68819:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2505.737763] LustreError: 68819:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2506.353229] LustreError: 68804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787616187 with bad export cookie 15895519286855951071 [ 2506.355816] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2506.363892] LustreError: 68804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2506.676606] Lustre: server umount lustre-MDT0001 complete [ 2520.438708] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2523.588531] Key type lgssc unregistered [ 2523.973561] LNet: 73693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2523.979778] LNetError: 73693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2524.014912] LNet: Removed LNI 192.168.201.156@tcp [ 2524.865150] Key type .llcrypt unregistered [ 2524.866942] Key type ._llcrypt unregistered [ 2550.193553] Key type ._llcrypt registered [ 2550.202078] Key type .llcrypt registered [ 2550.350124] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2551.244902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2551.295516] alg: No test for adler32 (adler32-zlib) [ 2552.437288] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 2552.703521] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2554.408276] Key type lgssc registered [ 2555.264670] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2564.731527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2564.763692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2566.554067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2570.508283] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2576.868175] LustreError: 75121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.900268] LustreError: 75121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2579.540106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2579.836299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2584.749992] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2593.164354] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2600.194660] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2607.445824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2607.748472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.854186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2614.538640] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2622.140128] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2631.045485] Lustre: Failing over lustre-MDT0000 [ 2631.136792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2631.141609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.163190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2631.320854] Lustre: server umount lustre-MDT0000 complete [ 2634.309190] LustreError: 75117:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2635.750663] LustreError: 75848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2635.751557] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2639.400437] LustreError: 76652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2639.422926] LustreError: 76652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2643.005555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2643.194898] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2643.473815] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2643.666031] LustreError: 78210:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2643.679901] LustreError: 78210:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.156@tcp: cfg command failed: rc = -9 [ 2643.704838] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2643.704838] [ 2643.717512] LustreError: MGC192.168.201.156@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2643.743054] LustreError: 78199:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2643.764759] LustreError: 78199:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -9 [ 2643.789166] LustreError: 78199:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 2643.798297] LustreError: 78199:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2643.827397] Lustre: server umount lustre-MDT0000 complete [ 2643.829429] LustreError: 78199:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2644.502148] LustreError: 75115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2644.544878] LustreError: 75115:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2650.388082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.491861] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2650.632415] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.56@tcp (not set up) [ 2650.775397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2650.849686] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2654.696671] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2655.844874] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2656.243318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2656.280785] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2656.322347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2666.469979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2666.472754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2666.473298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2666.482071] Lustre: Skipped 2 previous similar messages [ 2666.511179] Lustre: Skipped 1 previous similar message [ 2671.044853] Lustre: server umount lustre-OST0000 complete [ 2676.706904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2676.724535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2681.286429] Lustre: server umount lustre-MDT0000 complete [ 2681.825980] LustreError: 75120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.854268] LustreError: 75120:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2685.249076] LustreError: 77702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787616366 with bad export cookie 2125910850612333471 [ 2685.251721] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2685.265524] LustreError: 77702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2685.565523] Lustre: server umount lustre-MDT0001 complete [ 2695.348882] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2697.893639] Key type lgssc unregistered [ 2698.217685] LNet: 79992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2698.232606] LNetError: 79992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2699.311298] LNet: Removed LNI 192.168.201.156@tcp [ 2700.199408] Key type .llcrypt unregistered [ 2700.201214] Key type ._llcrypt unregistered [ 2712.316519] Key type ._llcrypt registered [ 2712.321180] Key type .llcrypt registered [ 2712.494742] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2712.899458] Key type .llcrypt unregistered [ 2712.904921] Key type ._llcrypt unregistered [ 2735.847720] Key type ._llcrypt registered [ 2735.849981] Key type .llcrypt registered [ 2736.000357] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2737.491955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2737.504055] alg: No test for adler32 (adler32-zlib) [ 2738.719231] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 2739.072421] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2740.808190] Key type lgssc registered [ 2742.253760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2749.708188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2749.720975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2751.229966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2756.568913] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2761.699181] LustreError: 81883:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.718982] LustreError: 81883:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2765.580699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.824066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2770.027875] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2777.263563] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2783.609300] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2791.710511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2792.200533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2796.332282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:449) [ 2799.061808] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2808.894087] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2821.091665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2821.106685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2821.115592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2821.131841] Lustre: Skipped 1 previous similar message [ 2824.802724] Lustre: server umount lustre-OST0000 complete [ 2831.334312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.342402] Lustre: Skipped 1 previous similar message [ 2831.345971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2831.353617] Lustre: Skipped 2 previous similar messages [ 2836.450293] LustreError: 82920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2836.457547] LustreError: 82920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2838.271252] Lustre: server umount lustre-MDT0000 complete [ 2841.576622] LustreError: 81884:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2841.585351] LustreError: 81884:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2841.651626] LustreError: 81864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787616522 with bad export cookie 7085642335620599069 [ 2841.656311] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2841.670489] LustreError: 81864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2841.893302] Lustre: server umount lustre-MDT0001 complete [ 2854.153098] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 2860.786142] Key type lgssc unregistered [ 2861.150964] LNet: 85422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2861.198131] LNetError: 85422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2861.242270] LNet: Removed LNI 192.168.201.156@tcp [ 2863.725732] Key type .llcrypt unregistered [ 2863.731205] Key type ._llcrypt unregistered [ 2897.819923] Key type ._llcrypt registered [ 2897.822941] Key type .llcrypt registered [ 2897.904358] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 2898.937224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2899.002099] alg: No test for adler32 (adler32-zlib) [ 2900.012707] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 2900.207983] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 2901.888237] Key type lgssc registered [ 2902.857406] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2912.605925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2912.616462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2913.933167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2917.985347] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2923.501081] LustreError: 86838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2923.548504] LustreError: 86838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2928.624732] LustreError: 86839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2930.687234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.521261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2937.339697] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2946.525176] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2953.486160] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2961.819051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2962.408920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.870292] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 2976.697700] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2983.925391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:481) [ 2994.148464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2994.150033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2994.163410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2994.164696] Lustre: Skipped 1 previous similar message [ 2994.175838] Lustre: Skipped 1 previous similar message [ 2999.277704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2999.980218] Lustre: server umount lustre-OST0000 complete [ 3007.972455] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3007.978387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.989483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3007.991403] Lustre: Skipped 1 previous similar message [ 3009.519282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3011.026976] Lustre: server umount lustre-MDT0000 complete [ 3014.632379] LustreError: 86839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3014.653170] LustreError: 86839:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3014.922064] LustreError: 86819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787616695 with bad export cookie 5743528120546842140 [ 3014.934536] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3015.253396] Lustre: server umount lustre-MDT0001 complete [ 3028.825499] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 3032.044506] Key type lgssc unregistered [ 3032.320510] LNet: 90400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3032.332100] LNetError: 90400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3032.359732] LNet: Removed LNI 192.168.201.156@tcp [ 3033.266255] Key type .llcrypt unregistered [ 3033.268313] Key type ._llcrypt unregistered [ 3057.618964] Key type ._llcrypt registered [ 3057.622721] Key type .llcrypt registered [ 3057.739338] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3058.819968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3059.159088] alg: No test for adler32 (adler32-zlib) [ 3060.332699] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 3060.599968] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 3062.289407] Key type lgssc registered [ 3063.481218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3071.838918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3071.871029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3073.505927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3078.472856] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3083.760423] LustreError: 91823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3083.785093] LustreError: 91823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3088.247620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3088.563039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3093.496972] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3102.757417] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3108.899817] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3114.820603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3115.181731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3118.261759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:451 to 0x280000401:513) [ 3120.820917] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3130.854033] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3149.281416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3149.292840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3149.295718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3149.315318] Lustre: Skipped 1 previous similar message [ 3149.331503] Lustre: Skipped 1 previous similar message [ 3153.408487] Lustre: server umount lustre-OST0000 complete [ 3159.522818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3159.548179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3163.726687] Lustre: server umount lustre-MDT0000 complete [ 3164.642266] LustreError: 91820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3164.660551] LustreError: 91820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3168.768678] LustreError: 91805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787616849 with bad export cookie 749127708449783391 [ 3168.770826] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3168.775651] LustreError: 91805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3169.150533] Lustre: server umount lustre-MDT0001 complete [ 3182.720852] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 3186.537479] Key type lgssc unregistered [ 3186.955740] LNet: 95380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3186.966422] LNetError: 95380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3186.992507] LNet: Removed LNI 192.168.201.156@tcp [ 3188.247837] Key type .llcrypt unregistered [ 3188.249447] Key type ._llcrypt unregistered [ 3217.212681] Key type ._llcrypt registered [ 3217.217465] Key type .llcrypt registered [ 3217.392969] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3219.092840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3219.157742] alg: No test for adler32 (adler32-zlib) [ 3220.219071] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 3220.422417] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 3222.128258] Key type lgssc registered [ 3223.583692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3232.308071] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3232.346668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3247.968524] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3253.088406] LustreError: 96768:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 3279.840437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3285.295255] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3294.855778] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3301.499462] Lustre: server umount lustre-OST0000 complete [ 3310.902742] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 20:16:30 (1787616990) [ 3319.113888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.328389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3324.372978] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3324.393640] LustreError: 98408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3324.415026] LustreError: 98408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3329.514914] LustreError: 98409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3332.988103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3333.535272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3338.202290] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3346.220323] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3352.393950] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3359.165892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3359.593855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3364.748633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:545) [ 3366.278696] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3374.547576] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3389.431659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3389.446444] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3389.463207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3389.474736] Lustre: Skipped 1 previous similar message [ 3393.792617] Lustre: server umount lustre-OST0000 complete [ 3399.653430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3399.667641] Lustre: Skipped 1 previous similar message [ 3399.672225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3399.684200] Lustre: Skipped 1 previous similar message [ 3404.140858] Lustre: server umount lustre-MDT0000 complete [ 3404.771334] LustreError: 98408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.795958] LustreError: 98408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3408.332621] LustreError: 98388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787617089 with bad export cookie 16563136210839960734 [ 3408.335102] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3408.340848] LustreError: 98388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3408.781958] Lustre: server umount lustre-MDT0001 complete [ 3419.483640] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 3422.364168] Key type lgssc unregistered [ 3422.736471] LNet: 102104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3422.750527] LNetError: 102104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3422.779905] LNet: Removed LNI 192.168.201.156@tcp [ 3423.601115] Key type .llcrypt unregistered [ 3423.603574] Key type ._llcrypt unregistered [ 3468.755219] Key type ._llcrypt registered [ 3468.757788] Key type .llcrypt registered [ 3468.892599] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3470.529925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3470.562960] alg: No test for adler32 (adler32-zlib) [ 3471.731902] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 3472.095724] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 3473.876465] Key type lgssc registered [ 3474.866468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3483.118817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3483.154772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3484.761824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3489.871282] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3494.887684] LustreError: 104291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3494.905988] LustreError: 104291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3499.229095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3499.651668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3504.775538] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3514.352949] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3520.834956] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3528.172280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3528.479345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3529.545949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:516 to 0x280000401:577) [ 3534.739247] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3545.067234] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3560.424902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3560.438431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3560.445056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3565.573167] LustreError: 106114:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3565.859601] Lustre: server umount lustre-OST0000 complete [ 3571.176166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3571.182820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3571.191074] Lustre: Skipped 1 previous similar message [ 3571.195126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3571.198907] Lustre: Skipped 1 previous similar message [ 3575.778213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3575.779317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3577.485266] Lustre: server umount lustre-MDT0000 complete [ 3580.902081] LustreError: 105292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3580.914463] LustreError: 105292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3581.332744] LustreError: 104271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787617262 with bad export cookie 9637531549775836655 [ 3581.339221] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3581.346630] LustreError: 104271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3581.528636] Lustre: server umount lustre-MDT0001 complete [ 3594.557715] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 3598.003871] Key type lgssc unregistered [ 3598.397989] LNet: 107850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3598.422202] LNetError: 107850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3598.441381] LNet: Removed LNI 192.168.201.156@tcp [ 3599.608216] Key type .llcrypt unregistered [ 3599.611416] Key type ._llcrypt unregistered [ 3624.519562] Key type ._llcrypt registered [ 3624.524857] Key type .llcrypt registered [ 3624.644800] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3625.544377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3625.559299] alg: No test for adler32 (adler32-zlib) [ 3626.582280] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 3626.781168] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 3628.432315] Key type lgssc registered [ 3629.807955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3638.361725] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3638.398994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3653.984976] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3659.104318] LustreError: 109095:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 3685.792307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3693.804397] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3702.672024] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3710.632516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.068330] LustreError: MGC192.168.201.156@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3711.083447] LustreError: 110120:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3711.102623] LustreError: 110120:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 3711.118658] LustreError: 110120:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 3711.127129] LustreError: 109101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787617391 with bad export cookie 7442038641703951957 [ 3711.134113] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3711.271764] Lustre: server umount lustre-MDT0000 complete [ 3711.278585] LustreError: 110120:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3736.773914] Lustre: server umount lustre-OST0000 complete [ 3748.098865] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 3757.127688] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3795.965538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3796.237727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3796.270764] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3796.362954] Lustre: lustre-MDT0000: new disk, initializing [ 3796.428334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.447944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3801.575202] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3813.174423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3813.399485] Lustre: 112910:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3813.468265] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3813.474088] Lustre: Skipped 1 previous similar message [ 3813.550940] Lustre: lustre-MDT0001: new disk, initializing [ 3813.622416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3813.658225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3813.671543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3818.126430] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3827.852233] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3833.176199] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3841.478680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3841.899917] Lustre: lustre-OST0000: new disk, initializing [ 3841.903792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3841.915517] Lustre: 114879:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3842.056260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3848.689949] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3850.804083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3850.813035] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3850.917176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3857.744196] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3862.421180] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3862.562833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3867.348563] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3867.548280] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3871.717622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3871.722134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3871.734344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3871.762542] Lustre: Skipped 1 previous similar message [ 3875.664572] Lustre: server umount lustre-OST0000 complete [ 3880.419198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3880.432945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3880.474894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3880.477300] Lustre: Skipped 1 previous similar message [ 3881.956819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3881.958826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3887.075554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3887.087271] Lustre: Skipped 2 previous similar messages [ 3891.177293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3891.182479] Lustre: Skipped 1 previous similar message [ 3894.240679] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3894.594321] Lustre: server umount lustre-MDT0000 complete [ 3898.125702] LustreError: 112904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787617578 with bad export cookie 7442038641703952062 [ 3898.133360] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3898.142868] LustreError: 112904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3898.579376] Lustre: server umount lustre-MDT0001 complete [ 3906.661142] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 20:26:25 (1787617585) [ 3910.410184] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3927.296265] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 3936.719980] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 3973.909508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3974.344277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3974.394809] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3974.520779] Lustre: lustre-MDT0000: new disk, initializing [ 3974.586854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3974.595681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3980.498493] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 3992.649216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3992.805043] Lustre: 119782:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3992.876706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3992.879732] Lustre: Skipped 1 previous similar message [ 3992.952844] Lustre: lustre-MDT0001: new disk, initializing [ 3993.038723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3993.087459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3993.096677] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3998.530790] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4007.992578] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4013.996758] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4020.718220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4021.020319] Lustre: lustre-OST0000: new disk, initializing [ 4021.031211] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4021.036155] Lustre: 121752:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4021.119968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4027.396604] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4027.473493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4027.481823] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4027.647982] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4036.969047] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4042.231369] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4042.518855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4047.500667] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4047.689595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4053.484562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4053.486720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4053.507355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4053.513559] Lustre: Skipped 1 previous similar message [ 4053.522035] Lustre: Skipped 1 previous similar message [ 4056.572567] Lustre: server umount lustre-OST0000 complete [ 4063.713675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4066.738879] Lustre: server umount lustre-MDT0000 complete [ 4068.839841] LustreError: 120807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.865078] LustreError: 120807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4070.296889] LustreError: 120576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787617751 with bad export cookie 7442038641703952944 [ 4070.309079] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4070.310177] LustreError: 120576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4070.641580] Lustre: server umount lustre-MDT0001 complete [ 4078.308729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4078.610379] LustreError: 123968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4078.660510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4082.559777] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4083.687958] LustreError: 123969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4088.806934] LustreError: 123968:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4093.384111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4099.231010] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4110.047547] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4116.155765] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4123.078186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4130.337746] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4138.127530] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4147.953930] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4155.361776] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4155.372967] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4155.374709] LustreError: Skipped 1 previous similar message [ 4155.376523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4155.376530] Lustre: Skipped 2 previous similar messages [ 4155.419377] Lustre: Skipped 2 previous similar messages [ 4158.196429] Lustre: server umount lustre-OST0000 complete [ 4165.603717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4165.615281] LustreError: Skipped 1 previous similar message [ 4176.352641] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4176.539237] Lustre: server umount lustre-MDT0000 complete [ 4180.241340] LustreError: 123951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787617861 with bad export cookie 7442038641703953826 [ 4180.254249] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4180.254380] LustreError: 123951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4180.558473] Lustre: server umount lustre-MDT0001 complete [ 4191.218190] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 4194.043743] Key type lgssc unregistered [ 4194.317689] LNet: 127601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4194.335557] LNetError: 127601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4194.364302] LNet: Removed LNI 192.168.201.156@tcp [ 4195.286565] Key type .llcrypt unregistered [ 4195.288611] Key type ._llcrypt unregistered [ 4212.700826] Key type ._llcrypt registered [ 4212.709104] Key type .llcrypt registered [ 4212.877439] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 4228.582643] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 4229.110769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4229.126293] alg: No test for adler32 (adler32-zlib) [ 4230.332426] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 4230.551681] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 4232.280192] Key type lgssc registered [ 4233.374100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4271.447803] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4271.468374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4272.830180] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4272.863841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4272.961236] Lustre: lustre-MDT0000: new disk, initializing [ 4273.037135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4273.053791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4277.479468] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4289.665589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4289.877574] Lustre: 130769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4290.056585] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4290.065078] Lustre: Skipped 1 previous similar message [ 4290.200678] Lustre: lustre-MDT0001: new disk, initializing [ 4290.388450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4290.439057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4290.452443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4295.481156] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4306.222480] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4312.154800] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4318.728887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4318.942226] Lustre: lustre-OST0000: new disk, initializing [ 4318.946777] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4318.950475] Lustre: 132738:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4319.006390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4326.023751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4326.038752] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4326.166618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4326.508626] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4339.253552] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4345.269644] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4345.477476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4352.133764] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4352.439396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4357.103537] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4357.119440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4357.128125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4357.140109] Lustre: Skipped 1 previous similar message [ 4357.152880] Lustre: Skipped 1 previous similar message [ 4360.818806] Lustre: server umount lustre-OST0000 complete [ 4367.329838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4367.341186] LustreError: Skipped 1 previous similar message [ 4367.346151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4367.358421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4367.379694] Lustre: Skipped 1 previous similar message [ 4367.388697] Lustre: Skipped 1 previous similar message [ 4372.452825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4377.570528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4377.586694] Lustre: Skipped 2 previous similar messages [ 4379.104492] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4379.573824] Lustre: server umount lustre-MDT0000 complete [ 4382.693078] LustreError: 130782:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4382.734261] LustreError: 130782:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4385.256190] LustreError: 130762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787618066 with bad export cookie 14168738104684473206 [ 4385.258927] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4385.283192] LustreError: 130762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4385.959894] Lustre: server umount lustre-MDT0001 complete [ 4396.225371] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 20:34:35 (1787618075) [ 4402.782399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4403.158456] LustreError: 135153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4403.222511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4407.626660] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4408.294485] LustreError: 135154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4413.410345] LustreError: 135153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4416.582609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4416.995340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4421.767407] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4429.732306] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4435.218343] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4437.474208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4437.478638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4437.488273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4443.330272] Lustre: server umount lustre-MDT0000 complete [ 4446.700120] LustreError: 135150:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4446.721742] LustreError: 135150:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4447.559957] LustreError: 135135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787618128 with bad export cookie 14168738104684474088 [ 4447.567832] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4447.572647] LustreError: 135135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4447.941843] Lustre: server umount lustre-MDT0001 complete [ 4458.421188] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 20:35:37 (1787618137) [ 4464.828867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4480.418585] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4485.536882] LustreError: 137569:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 4511.265163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4520.086397] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4528.553184] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4530.881450] Lustre: server umount lustre-OST0000 complete [ 4541.125696] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 20:37:00 (1787618220) [ 4548.606902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4549.265438] LustreError: 139015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4549.371662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4554.150919] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4562.878607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4567.707767] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4575.016989] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4580.941554] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4587.345017] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4587.760669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4587.773058] Lustre: Skipped 1 previous similar message [ 4594.771688] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4602.408486] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4614.626787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4614.642236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4614.666119] Lustre: Skipped 1 previous similar message [ 4614.684453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4614.693422] Lustre: Skipped 4 previous similar messages [ 4617.074772] Lustre: server umount lustre-MDT0000 complete [ 4620.781831] LustreError: 139016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4620.801526] LustreError: 139016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4621.170151] LustreError: 138998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787618302 with bad export cookie 14168738104684474753 [ 4621.187424] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4621.193234] LustreError: 138998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4621.485278] Lustre: server umount lustre-MDT0001 complete [ 4636.026321] Lustre: server umount lustre-OST0000 complete [ 4648.417690] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 20:38:46 (1787618326) [ 4657.041442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4657.743786] LustreError: 142673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4657.895098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4662.855819] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4673.401588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4677.811578] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4686.675548] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4693.345809] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4702.850885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4708.840888] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4718.424854] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4722.981735] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4727.693028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4728.547590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 4732.709194] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4732.985335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4738.017648] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4738.033315] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4738.051905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4738.063270] Lustre: Skipped 3 previous similar messages [ 4738.071997] Lustre: Skipped 3 previous similar messages [ 4741.824163] Lustre: server umount lustre-OST0000 complete [ 4748.257431] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4748.284854] Lustre: Skipped 1 previous similar message [ 4752.619329] Lustre: server umount lustre-MDT0000 complete [ 4753.390477] LustreError: 142674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4753.433296] LustreError: 142674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4757.932229] LustreError: 142655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787618438 with bad export cookie 14168738104684475817 [ 4757.933812] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4757.944258] LustreError: 142655:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4758.314605] Lustre: server umount lustre-MDT0001 complete [ 4768.351478] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 20:40:47 (1787618447) [ 4777.269532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4792.865165] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4797.985108] LustreError: 146876:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 4823.712488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4823.715794] Lustre: Skipped 2 previous similar messages [ 4830.284871] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4838.774654] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4846.054986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4846.556495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4851.383505] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4861.359523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4866.552926] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 4876.170126] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4884.549607] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4892.392972] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4892.816712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4898.127451] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4898.252765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4902.914270] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4902.916040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4902.932568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4902.932577] Lustre: Skipped 3 previous similar messages [ 4906.628989] Lustre: server umount lustre-MDT0000 complete [ 4908.008487] LustreError: 147915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4908.034856] LustreError: 147915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 4910.673316] LustreError: 146884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787618591 with bad export cookie 14168738104684476846 [ 4910.682652] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4910.701652] LustreError: 146884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4911.120591] Lustre: server umount lustre-MDT0001 complete [ 4925.837340] Lustre: server umount lustre-OST0000 complete [ 4934.303230] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 20:43:33 (1787618613) [ 4942.271466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4957.792842] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4962.912500] LustreError: 151076:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 4997.408727] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5006.058769] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5015.271033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5016.006735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5020.598345] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5029.257044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5034.499967] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5042.235276] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5047.921409] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5053.824369] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5054.013510] Lustre: lustre-OST0001: new disk, initializing [ 5054.020623] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5054.027412] Lustre: 153946:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5058.802978] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5060.133068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5060.141139] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5060.281485] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5067.604218] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5072.959250] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5073.323338] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5080.856270] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5081.013766] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5086.190351] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5086.192368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5086.212029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5086.212037] Lustre: Skipped 2 previous similar messages [ 5086.235943] Lustre: Skipped 3 previous similar messages [ 5090.404314] Lustre: server umount lustre-OST0000 complete [ 5100.337874] Lustre: server umount lustre-OST0001 complete [ 5111.880587] Lustre: server umount lustre-MDT0000 complete [ 5117.390265] LustreError: 151082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787618798 with bad export cookie 14168738104684477798 [ 5117.397680] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5117.411508] LustreError: 151082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5117.878668] Lustre: server umount lustre-MDT0001 complete [ 5147.867628] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 20:47:07 (1787618827) [ 5149.940307] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5151.663577] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5153.737377] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 20:47:12 (1787618832) [ 5161.484188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5161.947487] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5162.009595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5162.384182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5162.394391] Lustre: Skipped 6 previous similar messages [ 5166.789592] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5175.060411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5175.145732] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5175.167273] Lustre: 157509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5179.688176] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5187.774861] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5193.790732] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5200.824927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5200.960280] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5206.913894] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5210.802516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5214.993783] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5220.891869] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5221.129343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5228.972338] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5229.276951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5236.708031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5236.712117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5236.713106] LustreError: Skipped 3 previous similar messages [ 5236.719136] Lustre: Skipped 5 previous similar messages [ 5236.724415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5236.726075] Lustre: Skipped 7 previous similar messages [ 5238.498088] Lustre: server umount lustre-OST0000 complete [ 5264.206876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5272.178194] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5276.869237] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.894360] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5276.896509] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5276.928862] Lustre: Skipped 1 previous similar message [ 5280.079541] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5287.585391] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5287.867896] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5291.494880] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5291.723432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5298.053473] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9aa704b26800.ost_server_uuid 50 [ 5300.216655] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9aa704b26800.ost_server_uuid in FULL state after 0 sec [ 5300.373813] LustreError: 160778:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.56@tcp) returned error from blocking AST (req@00000000eb9540da x1874453238917376 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 00000000c43b3f72/0xc4a1783cad4286c0 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.56@tcp remote: 0xd15cef039f9966cd expref: 5 pid: 157515 timeout: 5400 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5300.428802] LustreError: lustre-MDT0000: A client on nid 192.168.201.56@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5300.437339] LustreError: 157505:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.56@tcp ns: mdt-lustre-MDT0000_UUID lock: 00000000c43b3f72/0xc4a1783cad4286c0 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.56@tcp remote: 0xd15cef039f9966cd expref: 6 pid: 157515 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5305.828655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5312.129607] Lustre: server umount lustre-OST0000 complete [ 5322.317027] Lustre: server umount lustre-MDT0000 complete [ 5323.232962] LustreError: 157521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5323.241065] LustreError: 157521:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [ 5326.319141] LustreError: 157501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619007 with bad export cookie 14168738104684478981 [ 5326.320824] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5326.345611] LustreError: 157501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5332.671739] Lustre: server umount lustre-MDT0001 complete [ 5345.543268] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 5349.985875] Key type lgssc unregistered [ 5350.493252] LNet: 163629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5350.518820] LNetError: 163629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5351.593606] LNet: Removed LNI 192.168.201.156@tcp [ 5352.929675] Key type .llcrypt unregistered [ 5352.937087] Key type ._llcrypt unregistered [ 5380.788526] Key type ._llcrypt registered [ 5380.798871] Key type .llcrypt registered [ 5381.016348] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 5382.219304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5382.292140] alg: No test for adler32 (adler32-zlib) [ 5383.401666] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 5383.716724] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 5385.368294] Key type lgssc registered [ 5386.845608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5395.578504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5395.604469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5397.173105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5403.068414] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5407.726807] LustreError: 165044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5407.752917] LustreError: 165044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5412.869594] LustreError: 165043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5413.804228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5414.852868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5419.508161] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5431.631459] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5439.111293] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5447.907927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5448.234552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5454.935226] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5463.599884] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5467.702071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5474.684306] Lustre: Failing over lustre-MDT0000 [ 5474.954164] Lustre: server umount lustre-MDT0000 complete [ 5475.907170] LustreError: 167346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5476.331504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5476.337464] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5477.860563] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5482.998972] LustreError: 165044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5483.042671] LustreError: 165044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 5487.147434] LustreError: 165024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.56@tcp arrived at 1787619168 with bad export cookie 7123130350991605363 [ 5487.172337] LustreError: 165024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5491.171251] LustreError: 165044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5491.202724] LustreError: 165044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 5493.236222] Lustre: 164355:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787619158/real 1787619158] req@000000006b7a8232 x1874454447683968/t0(0) o400->MGC192.168.201.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787619174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5493.287521] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5497.882233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5503.473888] LustreError: 167346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5503.486851] LustreError: 167346:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 5503.832793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5503.951980] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5509.095801] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5509.107983] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5509.537913] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5512.404344] Lustre: lustre-MDT0000: Denying connection for new client 8728a02f-99fc-4cc7-a4c3-3330b1eb9fa2 (at 192.168.201.56@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5524.453535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5524.480564] Lustre: Skipped 1 previous similar message [ 5524.489124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5524.505203] Lustre: Skipped 1 previous similar message [ 5529.592595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5529.602020] Lustre: Skipped 1 previous similar message [ 5534.693283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5534.706148] Lustre: Skipped 1 previous similar message [ 5536.224525] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5536.460649] Lustre: server umount lustre-OST0000 complete [ 5540.732178] LustreError: 168782:0:(ldlm_lib.c:3029:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5540.752813] Lustre: 168130:0:(ldlm_lib.c:2429:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5540.772806] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5540.823838] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5540.840386] LustreError: 168130:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001d7d437c x1874454447706624/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5540.864703] LustreError: 168130:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5540.884398] LustreError: 168130:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5540.898371] Lustre: lustre-MDT0000: Recovery over after 0:31, of 2 clients 0 recovered and 2 were evicted. [ 5541.443950] Lustre: server umount lustre-MDT0000 complete [ 5544.939024] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5544.951187] Lustre: Skipped 1 previous similar message [ 5544.998825] LustreError: 165039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5545.841513] LustreError: 165025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619226 with bad export cookie 7123130350991605692 [ 5545.845482] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5545.849451] LustreError: 165025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5545.849622] LustreError: 164352:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5546.085187] Lustre: server umount lustre-MDT0001 complete [ 5561.162295] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 5565.196477] Key type lgssc unregistered [ 5565.677715] LNet: 169538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5565.691991] LNetError: 169538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5565.728398] LNet: Removed LNI 192.168.201.156@tcp [ 5566.926185] Key type .llcrypt unregistered [ 5566.928317] Key type ._llcrypt unregistered [ 5591.445779] Key type ._llcrypt registered [ 5591.452696] Key type .llcrypt registered [ 5591.653706] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 5593.164692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5593.264986] alg: No test for adler32 (adler32-zlib) [ 5594.359653] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 5594.911733] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 5596.664369] Key type lgssc registered [ 5597.754475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5604.940536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5604.989256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5606.538746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5611.260814] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5617.130175] LustreError: 170951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5617.152177] LustreError: 170951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5620.124413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5620.647063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5625.316509] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5633.210486] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5639.300453] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5645.585814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5645.909848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5652.103489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5652.225613] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5660.456862] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5667.809687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5667.817627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5667.838885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5670.881304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5670.891580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5670.894765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5670.911690] Lustre: Skipped 1 previous similar message [ 5673.754710] Lustre: server umount lustre-OST0000 complete [ 5681.125067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5681.128622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5681.137241] Lustre: Skipped 1 previous similar message [ 5681.145887] Lustre: Skipped 1 previous similar message [ 5686.243215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5686.246373] Lustre: Skipped 1 previous similar message [ 5690.339897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5692.384825] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5692.692632] Lustre: server umount lustre-MDT0000 complete [ 5695.463284] LustreError: 170948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5695.480144] LustreError: 170948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5697.041200] LustreError: 170934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619377 with bad export cookie 17392811464104027607 [ 5697.063238] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5697.343360] Lustre: server umount lustre-MDT0001 complete [ 5712.745184] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 5715.913171] Key type lgssc unregistered [ 5716.227329] LNet: 174489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5716.251886] LNetError: 174489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5716.276173] LNet: Removed LNI 192.168.201.156@tcp [ 5717.187558] Key type .llcrypt unregistered [ 5717.192802] Key type ._llcrypt unregistered [ 5752.342197] Key type ._llcrypt registered [ 5752.350598] Key type .llcrypt registered [ 5752.501858] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 5754.073815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5754.092779] alg: No test for adler32 (adler32-zlib) [ 5755.313306] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 5755.554074] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 5757.200179] Key type lgssc registered [ 5758.427239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5765.347763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5765.371573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5767.084765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5772.303975] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5776.360331] LustreError: 176189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5776.389614] LustreError: 176189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5781.270628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5781.477483] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5781.625279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5786.596631] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5793.589539] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5799.339626] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5807.115443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5807.711638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5811.826572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5813.782356] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5821.597958] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5835.749110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5835.769140] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5835.801073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5836.769740] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5836.775951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5841.412805] Lustre: server umount lustre-OST0000 complete [ 5847.010876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5847.017161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5847.019909] Lustre: Skipped 3 previous similar messages [ 5852.129797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5856.230236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5856.235089] Lustre: Skipped 2 previous similar messages [ 5859.296122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5859.560575] Lustre: server umount lustre-MDT0000 complete [ 5861.348875] LustreError: 176190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5861.360320] LustreError: 176190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5862.866091] LustreError: 176924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619543 with bad export cookie 9141581592620270265 [ 5862.867786] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5862.879776] LustreError: 176924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5863.461379] Lustre: server umount lustre-MDT0001 complete [ 5875.471048] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 5878.477889] Key type lgssc unregistered [ 5878.827533] LNet: 179750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5878.844911] LNetError: 179750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5879.910890] LNet: Removed LNI 192.168.201.156@tcp [ 5880.871256] Key type .llcrypt unregistered [ 5880.873316] Key type ._llcrypt unregistered [ 5902.629174] Key type ._llcrypt registered [ 5902.630972] Key type .llcrypt registered [ 5902.768229] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 5904.041588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5904.151520] alg: No test for adler32 (adler32-zlib) [ 5905.186696] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 5905.423464] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 5907.096181] Key type lgssc registered [ 5908.130510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5918.064825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5918.105294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5919.796354] Lustre: *** cfs_fail_loc=135, val=0*** [ 5919.897432] LustreError: 181275:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5919.905309] LustreError: 181275:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.156@tcp: cfg command failed: rc = -2 [ 5919.912048] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5919.912048] [ 5919.918981] LustreError: MGC192.168.201.156@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5919.940551] LustreError: 181236:0:(tgt_mount.c:1888:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5919.946878] LustreError: 181236:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 5919.956634] LustreError: 181236:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-MDT0000 [ 5919.964224] LustreError: 181236:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5920.029326] Lustre: server umount lustre-MDT0000 complete [ 5920.032979] LustreError: 181236:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5927.448715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5928.029151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5931.864423] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5938.153171] LustreError: 181647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5938.164227] LustreError: 181647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5939.761757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5940.132895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5943.911983] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 5951.364844] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5957.850415] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5960.673443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5960.680456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5960.689056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5963.746383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5963.747184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5963.757950] Lustre: Skipped 1 previous similar message [ 5964.775109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5964.783575] Lustre: Skipped 1 previous similar message [ 5966.015719] Lustre: server umount lustre-MDT0000 complete [ 5969.485393] LustreError: 181629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619650 with bad export cookie 6161882866671516655 [ 5969.494632] LustreError: 181629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5969.497077] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5969.806569] Lustre: server umount lustre-MDT0001 complete [ 5979.286628] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 5982.326953] Key type lgssc unregistered [ 5982.639939] LNet: 183982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5982.646058] LNetError: 183982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5982.667656] LNet: Removed LNI 192.168.201.156@tcp [ 5983.595744] Key type .llcrypt unregistered [ 5983.600133] Key type ._llcrypt unregistered [ 6006.370100] Key type ._llcrypt registered [ 6006.372521] Key type .llcrypt registered [ 6006.532795] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 6006.865331] Key type .llcrypt unregistered [ 6006.868194] Key type ._llcrypt unregistered [ 6025.725388] Key type ._llcrypt registered [ 6025.726740] Key type .llcrypt registered [ 6025.774568] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 6026.937438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6026.981410] alg: No test for adler32 (adler32-zlib) [ 6028.028095] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 6028.298491] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 6029.976527] Key type lgssc registered [ 6031.177043] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6038.715436] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6038.732451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6054.304710] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6059.425502] LustreError: 186120:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.156@tcp: failed processing log, type 4: rc = -110 [ 6086.176942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6092.338836] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6100.052450] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6107.992664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6108.586506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6108.676760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6113.761928] LustreError: 186146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6113.782184] LustreError: 186146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6113.926115] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6118.887608] LustreError: 186145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6120.936100] LustreError: 187153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6120.970356] LustreError: 187153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6123.175915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6123.781782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6128.890204] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6138.033730] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6144.875595] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6149.862139] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6164.972950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6164.985345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6164.999619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6165.009687] Lustre: Skipped 1 previous similar message [ 6165.031455] Lustre: Skipped 1 previous similar message [ 6170.104185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6170.119054] Lustre: Skipped 1 previous similar message [ 6171.007527] Lustre: server umount lustre-OST0000 complete [ 6180.328727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6180.339929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6180.340232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6180.348707] Lustre: Skipped 1 previous similar message [ 6180.380518] LustreError: Skipped 1 previous similar message [ 6182.119884] Lustre: server umount lustre-MDT0000 complete [ 6185.450960] LustreError: 187153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6185.488513] LustreError: 187153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6186.680885] LustreError: 186126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787619867 with bad export cookie 16065445717767322459 [ 6186.686741] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6186.696260] LustreError: 186126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6187.022255] Lustre: server umount lustre-MDT0001 complete [ 6198.900888] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 6202.385683] Key type lgssc unregistered [ 6202.705407] LNet: 190090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6202.712693] LNetError: 190090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6203.797512] LNet: Removed LNI 192.168.201.156@tcp [ 6204.707267] Key type .llcrypt unregistered [ 6204.710492] Key type ._llcrypt unregistered [ 6230.320732] Key type ._llcrypt registered [ 6230.324861] Key type .llcrypt registered [ 6230.517924] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 6231.867302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6231.916081] alg: No test for adler32 (adler32-zlib) [ 6233.392757] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 6233.645209] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 6235.304880] Key type lgssc registered [ 6237.098724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6247.091404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6247.109923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6248.887956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6258.153131] LustreError: 191505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6258.188077] LustreError: 191505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6258.446230] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6263.266424] LustreError: 191506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6268.389255] LustreError: 191505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6270.417563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6270.939722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6278.180711] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6289.521483] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6297.997990] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6306.185504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6306.795577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6314.307891] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6319.093886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6324.043938] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6329.966080] Lustre: Failing over lustre-MDT0000 [ 6330.452958] Lustre: server umount lustre-MDT0000 complete [ 6330.966779] LustreError: 191501:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6332.385799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6332.396179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6334.434349] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6334.460181] Lustre: Skipped 1 previous similar message [ 6336.031459] LustreError: 191500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.56@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6336.050843] LustreError: 191500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6344.688677] LustreError: 191505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6344.724571] LustreError: 191505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 6350.752561] Lustre: 190819:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787620015/real 1787620015] req@00000000b86287a5 x1874455338973824/t0(0) o400->MGC192.168.201.156@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787620031 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6350.820040] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6353.653887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6361.065263] LustreError: 191502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6361.070387] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x21476d09fdf9d940 [ 6361.086485] LustreError: 191502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 6361.124212] Lustre: MGC192.168.201.156@tcp: Connection restored to 0@lo (at 0@lo) [ 6361.326464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6361.417966] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6361.665900] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6366.695784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6366.714532] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6366.792210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6367.179946] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6375.250579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6385.873499] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6402.541094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6402.568287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6402.574808] Lustre: Skipped 1 previous similar message [ 6405.516987] Lustre: server umount lustre-OST0000 complete [ 6412.775351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6412.779514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6412.783149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6412.801193] Lustre: Skipped 2 previous similar messages [ 6416.500587] Lustre: server umount lustre-MDT0000 complete [ 6417.890922] LustreError: 191502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6417.906242] LustreError: 191502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6420.659597] LustreError: 191486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620101 with bad export cookie 2398005216281221440 [ 6420.661158] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6420.673787] LustreError: 191486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6421.191198] Lustre: server umount lustre-MDT0001 complete [ 6432.222823] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 6435.359282] Key type lgssc unregistered [ 6435.542863] LNet: 196427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6435.555380] LNetError: 196427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6435.575098] LNet: Removed LNI 192.168.201.156@tcp [ 6436.681613] Key type .llcrypt unregistered [ 6436.686270] Key type ._llcrypt unregistered [ 6470.683763] Key type ._llcrypt registered [ 6470.684899] Key type .llcrypt registered [ 6470.751410] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 6471.577193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6471.688851] alg: No test for adler32 (adler32-zlib) [ 6472.620396] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 6472.765741] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 6474.384168] Key type lgssc registered [ 6475.442336] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6484.238428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6484.252792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.921383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6491.793627] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6495.202580] LustreError: 197843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6495.219997] LustreError: 197843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6500.077178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6500.322020] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6500.771565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6506.478256] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6514.981223] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6522.039396] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6531.179226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6531.751079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6539.389352] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6548.340553] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6555.323458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6561.879033] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6572.885223] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6599.284536] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6611.939649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6611.942147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6611.975142] Lustre: Skipped 1 previous similar message [ 6611.998818] Lustre: Skipped 1 previous similar message [ 6615.975802] Lustre: server umount lustre-OST0000 complete [ 6623.719386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6623.740684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6623.789140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6627.300911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6627.339770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6627.373156] Lustre: Skipped 1 previous similar message [ 6629.482669] Lustre: server umount lustre-MDT0000 complete [ 6631.460386] LustreError: 197842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6631.493708] LustreError: 197842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6635.313583] LustreError: 197823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620316 with bad export cookie 15816480923801462974 [ 6635.317654] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6635.329294] LustreError: 197823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6635.972949] Lustre: server umount lustre-MDT0001 complete [ 6649.557787] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 6653.660753] Key type lgssc unregistered [ 6653.982147] LNet: 201755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6653.986697] LNetError: 201755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6654.007713] LNet: Removed LNI 192.168.201.156@tcp [ 6654.718809] Key type .llcrypt unregistered [ 6654.720534] Key type ._llcrypt unregistered [ 6672.291364] Key type ._llcrypt registered [ 6672.296647] Key type .llcrypt registered [ 6672.372851] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 6673.542956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6673.659695] alg: No test for adler32 (adler32-zlib) [ 6674.830535] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 6675.292686] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 6677.096210] Key type lgssc registered [ 6678.595049] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6686.412789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6686.428388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6687.729323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6691.726822] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6697.955820] LustreError: 203002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6697.966226] LustreError: 203002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6699.589767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6699.963515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6703.606772] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6711.277023] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6717.106487] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6724.824089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6725.535166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6732.668629] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6732.753095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6742.347882] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6749.184325] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6760.370267] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6770.635127] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6778.523769] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6791.137768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6791.161250] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6791.204181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6794.209566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6794.222653] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6794.265804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6794.269223] Lustre: Skipped 1 previous similar message [ 6795.596747] Lustre: server umount lustre-OST0000 complete [ 6802.401759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6802.410326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6802.428033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6804.449163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6804.454605] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6806.717397] Lustre: server umount lustre-MDT0000 complete [ 6809.569415] LustreError: 202999:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6809.594228] LustreError: 202999:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6812.811230] LustreError: 204848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620493 with bad export cookie 5316166120670564450 [ 6812.813721] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6812.820855] LustreError: 204848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6813.243106] Lustre: server umount lustre-MDT0001 complete [ 6827.214786] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 6830.929866] Key type lgssc unregistered [ 6831.218761] LNet: 207737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6831.223347] LNetError: 207737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6831.240050] LNet: Removed LNI 192.168.201.156@tcp [ 6833.057272] Key type .llcrypt unregistered [ 6833.059952] Key type ._llcrypt unregistered [ 6860.435414] Key type ._llcrypt registered [ 6860.436505] Key type .llcrypt registered [ 6860.586758] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 6862.218944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6862.270518] alg: No test for adler32 (adler32-zlib) [ 6863.235446] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 6863.398142] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 6865.032142] Key type lgssc registered [ 6865.930649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6876.484691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6876.507670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6878.125347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6885.738481] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6887.459789] LustreError: 209153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6887.479265] LustreError: 209153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6892.514708] LustreError: 209152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6897.640794] LustreError: 209153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6899.005218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6899.393414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6904.682265] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6913.254792] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6920.734844] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6927.658333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6928.240144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6942.227933] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 6947.259209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6953.554300] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6969.833817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6969.843010] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6969.867707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6972.902284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6972.906954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6972.907254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6975.421425] Lustre: server umount lustre-OST0000 complete [ 6982.128551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6982.136145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6982.146326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6982.150121] Lustre: Skipped 1 previous similar message [ 6988.264902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6988.275168] Lustre: Skipped 3 previous similar messages [ 6993.378066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6993.379394] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 6993.385201] Lustre: Skipped 1 previous similar message [ 6993.573042] Lustre: server umount lustre-MDT0000 complete [ 6997.477803] LustreError: 209153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6997.515163] LustreError: 209153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6997.722516] LustreError: 211886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620678 with bad export cookie 3013416631233936641 [ 6997.727969] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6997.731783] LustreError: 211886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6998.151441] Lustre: server umount lustre-MDT0001 complete [ 7013.760793] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 7017.442521] Key type lgssc unregistered [ 7017.707635] LNet: 212846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7017.714921] LNetError: 212846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7017.725814] LNet: Removed LNI 192.168.201.156@tcp [ 7018.583960] Key type .llcrypt unregistered [ 7018.586721] Key type ._llcrypt unregistered [ 7039.101673] Key type ._llcrypt registered [ 7039.104941] Key type .llcrypt registered [ 7039.228983] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7040.803613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7040.946587] alg: No test for adler32 (adler32-zlib) [ 7041.984382] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 7042.248441] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 7044.017078] Key type lgssc registered [ 7045.515242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7053.785181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7053.803531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7055.143287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7059.327410] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7064.547806] LustreError: 214112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7064.569197] LustreError: 214112:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7067.785768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7068.141579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7072.594238] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7079.656412] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7084.730115] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7091.506719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7091.967062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7098.148622] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7100.182545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 7105.945634] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7120.874704] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7120.879582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7120.891194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7125.829947] Lustre: server umount lustre-OST0000 complete [ 7131.106954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7131.120875] Lustre: Skipped 1 previous similar message [ 7131.123271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7131.132296] Lustre: Skipped 2 previous similar messages [ 7136.259159] LustreError: 214107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7136.282549] LustreError: 214107:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7136.513444] Lustre: server umount lustre-MDT0000 complete [ 7140.392035] LustreError: 214113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7140.414032] LustreError: 214113:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7141.415437] LustreError: 214094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787620822 with bad export cookie 5506478037876703910 [ 7141.434098] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7141.843361] Lustre: server umount lustre-MDT0001 complete [ 7154.854611] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 7158.889880] Key type lgssc unregistered [ 7159.469276] LNet: 217798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7159.487601] LNetError: 217798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7159.531902] LNet: Removed LNI 192.168.201.156@tcp [ 7161.308859] Key type .llcrypt unregistered [ 7161.315529] Key type ._llcrypt unregistered [ 7184.088451] Key type ._llcrypt registered [ 7184.089883] Key type .llcrypt registered [ 7184.205107] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7185.430301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7185.475876] alg: No test for adler32 (adler32-zlib) [ 7186.794630] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 7187.196244] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 7188.945204] Key type lgssc registered [ 7190.462986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7200.082733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7200.148143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7201.753048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7206.748743] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7212.002001] LustreError: 219064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7212.010348] LustreError: 219064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7216.207884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7216.929296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7221.765171] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7231.524898] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7238.946136] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7247.294029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7247.796348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7254.170737] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7262.425393] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7271.374256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7282.411228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7282.483646] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7282.673273] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7289.736896] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7300.142795] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7302.231863] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7314.874335] Lustre: Permanently deactivating lustre-OST0001 [ 7314.889883] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7322.251222] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7330.585810] Lustre: Permanently reactivating lustre-OST0001 [ 7330.647829] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7330.650673] Lustre: Skipped 2 previous similar messages [ 7339.487874] Lustre: server umount lustre-OST0001 complete [ 7340.601286] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7340.623428] LustreError: 222537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7344.098791] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7344.113835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7344.136671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7346.161195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7346.162369] LustreError: 220887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7346.174258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7350.426319] Lustre: server umount lustre-OST0000 complete [ 7354.849223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7354.866864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7354.893440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7354.897754] Lustre: Skipped 1 previous similar message [ 7360.992906] LustreError: 223025:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 7361.505797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7361.513880] Lustre: Skipped 3 previous similar messages [ 7366.629071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7366.637216] Lustre: Skipped 1 previous similar message [ 7369.184949] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7369.362827] Lustre: server umount lustre-MDT0000 complete [ 7371.750869] LustreError: 219060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7371.770502] LustreError: 219060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7373.606637] LustreError: 219045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621054 with bad export cookie 7728747124262673061 [ 7373.607955] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7373.630223] LustreError: 219045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7374.130357] Lustre: server umount lustre-MDT0001 complete [ 7387.385595] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 7390.883588] Key type lgssc unregistered [ 7391.207356] LNet: 224136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7391.218580] LNetError: 224136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7392.295814] LNet: Removed LNI 192.168.201.156@tcp [ 7393.417912] Key type .llcrypt unregistered [ 7393.421046] Key type ._llcrypt unregistered [ 7441.118342] Key type ._llcrypt registered [ 7441.120949] Key type .llcrypt registered [ 7441.248453] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7442.585858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7442.634690] alg: No test for adler32 (adler32-zlib) [ 7443.686193] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 7443.882164] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 7445.521579] Key type lgssc registered [ 7446.921687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7455.218636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7455.250717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7456.412385] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7456.424720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7456.739843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7460.672818] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7470.799537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7470.949789] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7471.022794] Lustre: 226341:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7471.426594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7476.971658] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7486.160243] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7492.860488] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7501.135642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7501.403878] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7501.695878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7508.598844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7509.075454] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7518.307742] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7521.307597] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7527.131534] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7534.996944] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7543.493438] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7558.062703] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7558.068481] Lustre: Skipped 1 previous similar message [ 7577.516331] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7577.528538] Lustre: Skipped 1 previous similar message [ 7614.684898] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7614.690414] Lustre: Skipped 4 previous similar messages [ 7682.380587] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 7682.387267] Lustre: Skipped 7 previous similar messages [ 7703.014583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7703.022509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7703.031112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7703.043086] Lustre: Skipped 1 previous similar message [ 7708.400361] Lustre: server umount lustre-OST0000 complete [ 7717.365114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7717.378950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7717.385322] Lustre: Skipped 1 previous similar message [ 7717.389696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7718.380368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7718.414976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7718.427131] Lustre: Skipped 1 previous similar message [ 7719.906393] Lustre: server umount lustre-MDT0000 complete [ 7723.492667] LustreError: 229925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7723.517981] LustreError: 229925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7725.652455] LustreError: 226332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621406 with bad export cookie 9821317010571332962 [ 7725.661676] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7725.669758] LustreError: 226332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7726.239733] Lustre: server umount lustre-MDT0001 complete [ 7745.357934] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 7748.548570] Key type lgssc unregistered [ 7748.940930] LNet: 231181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7748.952057] LNetError: 231181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7748.966730] LNet: Removed LNI 192.168.201.156@tcp [ 7750.008487] Key type .llcrypt unregistered [ 7750.012390] Key type ._llcrypt unregistered [ 7776.759361] Key type ._llcrypt registered [ 7776.761570] Key type .llcrypt registered [ 7776.860526] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7777.854916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7777.923733] alg: No test for adler32 (adler32-zlib) [ 7779.127258] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 7779.445506] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 7781.163260] Key type lgssc registered [ 7782.901478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7790.764860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7790.785771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7792.304648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7798.800651] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7802.849411] LustreError: 232595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7802.922577] LustreError: 232595:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7807.984414] LustreError: 232596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7810.843651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7811.374163] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7817.337322] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7827.444208] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7835.195295] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7843.968246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7844.317760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7851.697694] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_default_debug -1 all [ 7861.347407] Lustre: DEBUG MARKER: oleg156-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7862.838661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7893.474479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7893.477291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7893.480453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7893.487922] Lustre: Skipped 1 previous similar message [ 7894.562208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7894.576957] Lustre: Skipped 2 previous similar messages [ 7900.646025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7906.801639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7906.808042] Lustre: Skipped 2 previous similar messages [ 7907.296127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7907.417889] Lustre: server umount lustre-OST0000 complete [ 7913.185876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7913.198376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7913.218221] Lustre: Skipped 1 previous similar message [ 7918.810971] Lustre: server umount lustre-MDT0000 complete [ 7922.390367] LustreError: 232575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787621603 with bad export cookie 9318571704783082708 [ 7922.391502] LustreError: MGC192.168.201.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7922.399128] LustreError: 232575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7922.661603] Lustre: server umount lustre-MDT0001 complete [ 7936.139296] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 7939.405320] Key type lgssc unregistered [ 7939.669274] LNet: 236305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7939.676981] LNetError: 236305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7940.718243] LNet: Removed LNI 192.168.201.156@tcp [ 7941.901503] Key type .llcrypt unregistered [ 7941.902773] Key type ._llcrypt unregistered [ 7967.740119] Key type ._llcrypt registered [ 7967.741828] Key type .llcrypt registered [ 7967.886211] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 7968.831399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7968.855234] alg: No test for adler32 (adler32-zlib) [ 7969.949922] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 7970.152648] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 7971.808357] Key type lgssc registered [ 7973.865397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8249.661358] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 8253.959224] Key type lgssc unregistered [ 8254.277862] LNet: 238331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8254.287928] LNetError: 238331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8254.339438] LNet: Removed LNI 192.168.201.156@tcp [ 8255.330082] Key type .llcrypt unregistered [ 8255.332870] Key type ._llcrypt unregistered [ 8294.540183] Key type ._llcrypt registered [ 8294.543821] Key type .llcrypt registered [ 8294.780406] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing set_hostid [ 8317.236889] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing load_modules_local [ 8318.300301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8318.369839] alg: No test for adler32 (adler32-zlib) [ 8320.035847] Lustre: Lustre: Build Version: 2.17.57_80_ga3b037e [ 8320.695367] LNet: Added LNI 192.168.201.156@tcp [8/256/0/180] [ 8322.697267] Key type lgssc registered [ 8324.969971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8360.865537] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8016 sec ========== 21:40:39 (1787622039) [ 8362.850068] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:40:41 (1787622041) === [ 8367.918874] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:40:46 (1787622046) === [ 8424.336138] Lustre: DEBUG MARKER: oleg156-server.virtnet: executing unload_modules_local [ 8427.308715] Key type lgssc unregistered [ 8427.765052] LNet: 243577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8427.774725] LNetError: 243577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8427.788206] LNet: Removed LNI 192.168.201.156@tcp [ 8428.812827] Key type .llcrypt unregistered [ 8428.814540] Key type ._llcrypt unregistered