[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 507582847 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002283] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.006487] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.010051] LSM: Security Framework initializing [ 0.011064] Yama: becoming mindful. [ 0.012042] SELinux: Initializing. [ 0.013083] *** VALIDATE selinux *** [ 0.021515] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026807] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028058] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031071] *** VALIDATE tmpfs *** [ 0.032316] *** VALIDATE proc *** [ 0.033250] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.036176] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038193] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040030] Spectre V2 : User space: Vulnerable [ 0.041007] Speculative Store Bypass: Vulnerable [ 0.044201] debug: unmapping init [mem 0xffffffffaca59000-0xffffffffaca60fff] [ 0.046921] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047566] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048020] ... version: 2 [ 0.048650] ... bit width: 48 [ 0.049027] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.054280] rcu: Hierarchical SRCU implementation. [ 0.056595] smp: Bringing up secondary CPUs ... [ 0.057490] x86: Booting SMP configuration: [ 0.058028] .... node #0, CPUs: #1 #2 #3 [ 0.061255] smp: Brought up 1 node, 4 CPUs [ 0.063017] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136353] node 0 deferred pages initialised in 68ms [ 0.139112] devtmpfs: initialized [ 0.140303] x86/mm: Memory block size: 128MB [ 0.142861] gcov: version magic: 0x41383552 [ 0.143575] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145120] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147514] pinctrl core: initialized pinctrl subsystem [ 0.148156] [ 0.148432] ************************************************************* [ 0.150011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151007] ** ** [ 0.153012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.155010] ** ** [ 0.156008] ** This means that this kernel is built to expose internal ** [ 0.158008] ** IOMMU data structures, which may compromise security on ** [ 0.159012] ** your system. ** [ 0.161014] ** ** [ 0.163009] ** If you see this message and you are not debugging the ** [ 0.164009] ** kernel, report this immediately to your vendor! ** [ 0.166009] ** ** [ 0.167008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169009] ************************************************************* [ 0.170593] NET: Registered protocol family 16 [ 0.172369] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.173038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.177083] cpuidle: using governor menu [ 0.178468] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179347] PCI: Using configuration type 1 for base access [ 0.180096] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186114] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.187014] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.190035] cryptd: max_cpu_qlen set to 1000 [ 0.193186] ACPI: Added _OSI(Module Device) [ 0.194010] ACPI: Added _OSI(Processor Device) [ 0.194853] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.196010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.199346] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.203351] ACPI: Interpreter enabled [ 0.204043] ACPI: PM: (supports S0 S3 S4 S5) [ 0.204922] ACPI: Using IOAPIC for interrupt routing [ 0.206148] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.209389] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.217512] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.219036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.221020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.223068] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.227228] acpiphp: Slot [2] registered [ 0.228103] acpiphp: Slot [5] registered [ 0.228967] acpiphp: Slot [6] registered [ 0.230093] acpiphp: Slot [7] registered [ 0.231000] acpiphp: Slot [8] registered [ 0.232117] acpiphp: Slot [9] registered [ 0.233077] acpiphp: Slot [10] registered [ 0.233952] acpiphp: Slot [3] registered [ 0.235100] acpiphp: Slot [4] registered [ 0.237119] acpiphp: Slot [11] registered [ 0.237960] acpiphp: Slot [12] registered [ 0.238058] acpiphp: Slot [13] registered [ 0.239062] acpiphp: Slot [14] registered [ 0.241128] acpiphp: Slot [15] registered [ 0.242105] acpiphp: Slot [16] registered [ 0.243154] acpiphp: Slot [17] registered [ 0.244017] acpiphp: Slot [18] registered [ 0.244803] acpiphp: Slot [19] registered [ 0.245052] acpiphp: Slot [20] registered [ 0.246082] acpiphp: Slot [21] registered [ 0.246923] acpiphp: Slot [22] registered [ 0.248068] acpiphp: Slot [23] registered [ 0.248907] acpiphp: Slot [24] registered [ 0.250057] acpiphp: Slot [25] registered [ 0.250951] acpiphp: Slot [26] registered [ 0.251070] acpiphp: Slot [27] registered [ 0.251871] acpiphp: Slot [28] registered [ 0.253064] acpiphp: Slot [29] registered [ 0.253860] acpiphp: Slot [30] registered [ 0.255063] acpiphp: Slot [31] registered [ 0.255955] PCI host bridge to bus 0000:00 [ 0.256013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.257015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.259030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.260019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.262015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.263016] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.265225] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.266665] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.268801] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.275019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.278044] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.281021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.283021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.286020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.288586] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.291843] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.295041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.297693] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.301017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.312013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.316018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.319343] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.324013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.328015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.343014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.351727] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.356027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.363019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.381017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.391845] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.397022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.402021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.416021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.423657] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.430026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.435020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.456013] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.473192] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.478019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.484017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.498017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.506338] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.512020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.517019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.529019] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.540683] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.543381] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.545354] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.548362] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.551268] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.556093] iommu: Default domain type: Passthrough [ 0.557425] SCSI subsystem initialized [ 0.559084] ACPI: bus type USB registered [ 0.560061] usbcore: registered new interface driver usbfs [ 0.561052] usbcore: registered new interface driver hub [ 0.562053] usbcore: registered new device driver usb [ 0.563125] pps_core: LinuxPPS API ver. 1 registered [ 0.564007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.566068] PTP clock support registered [ 0.569037] EDAC MC: Ver: 3.0.0 [ 0.570084] PCI: Using ACPI for IRQ routing [ 0.572801] NetLabel: Initializing [ 0.573010] NetLabel: domain hash size = 128 [ 0.574010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.575063] NetLabel: unlabeled traffic allowed by default [ 0.576099] vgaarb: loaded [ 0.577215] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.579013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.585360] clocksource: Switched to clocksource kvm-clock [ 0.703267] VFS: Disk quotas dquot_6.6.0 [ 0.704937] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.706867] *** VALIDATE ramfs *** [ 0.708320] *** VALIDATE hugetlbfs *** [ 0.710700] pnp: PnP ACPI init [ 0.713330] pnp: PnP ACPI: found 6 devices [ 0.731041] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.733200] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.734563] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.736021] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.737819] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.739451] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.742263] NET: Registered protocol family 2 [ 0.744626] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.748721] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.751197] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.756254] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.758874] TCP: Hash tables configured (established 65536 bind 65536) [ 0.760752] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.762683] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.764643] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.766536] NET: Registered protocol family 1 [ 0.768326] RPC: Registered named UNIX socket transport module. [ 0.769984] RPC: Registered udp transport module. [ 0.771161] RPC: Registered tcp transport module. [ 0.772260] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.774016] NET: Registered protocol family 44 [ 0.775112] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.776390] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.777691] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.779414] PCI: CLS 0 bytes, default 64 [ 0.780659] Unpacking initramfs... [ 2.119247] debug: unmapping init [mem 0xffff956c3cc54000-0xffff956c3ffbffff] [ 2.122866] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.124940] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.127556] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.638430] Initialise system trusted keyrings [ 2.640495] Key type blacklist registered [ 2.642454] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.652540] zbud: loaded [ 2.656828] *** VALIDATE nfs *** [ 2.657804] *** VALIDATE nfs4 *** [ 2.658983] pstore: using deflate compression [ 2.663070] Platform Keyring initialized [ 2.773233] NET: Registered protocol family 38 [ 2.774521] Key type asymmetric registered [ 2.775700] Asymmetric key parser 'x509' registered [ 2.777100] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.779436] io scheduler mq-deadline registered [ 2.780668] io scheduler kyber registered [ 2.781871] io scheduler bfq registered [ 2.784178] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.787087] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.790016] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.792793] ACPI: Power Button [PWRF] [ 2.797546] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.802192] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.813778] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.822737] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.839292] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.866625] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.893688] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.897418] Non-volatile memory driver v1.3 [ 2.898910] Linux agpgart interface v0.103 [ 2.930539] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.933169] vda: detected capacity change from 0 to 74698752 [ 2.952380] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.954159] vdb: detected capacity change from 0 to 1073741824 [ 2.966356] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.968271] vdc: detected capacity change from 0 to 2621440000 [ 2.981235] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.983067] vdd: detected capacity change from 0 to 2621440000 [ 2.995786] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.997730] vde: detected capacity change from 0 to 4294967296 [ 3.010774] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.012546] vdf: detected capacity change from 0 to 4294967296 [ 3.017558] libphy: Fixed MDIO Bus: probed [ 3.024955] usbcore: registered new interface driver usbserial_generic [ 3.027921] usbserial: USB Serial support registered for generic [ 3.030698] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.035487] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.037662] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.040211] mousedev: PS/2 mouse device common for all mice [ 3.042372] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.045494] rtc_cmos 00:05: RTC can wake from S4 [ 3.047555] rtc_cmos 00:05: registered as rtc0 [ 3.049821] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.053645] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.056952] intel_pstate: CPU model not supported [ 3.057826] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.063599] hid: raw HID events driver (C) Jiri Kosina [ 3.065845] usbcore: registered new interface driver usbhid [ 3.068020] usbhid: USB HID core driver [ 3.069682] drop_monitor: Initializing network drop monitor service [ 3.071957] Initializing XFRM netlink socket [ 3.073947] NET: Registered protocol family 10 [ 3.077647] Segment Routing with IPv6 [ 3.080094] NET: Registered protocol family 17 [ 3.081735] mpls_gso: MPLS GSO support [ 3.087808] RAS: Correctable Errors collector initialized. [ 3.090280] AVX version of gcm_enc/dec engaged. [ 3.091959] AES CTR mode by8 optimization enabled [ 3.149258] sched_clock: Marking stable (3149238940, 0)->(4011880484, -862641544) [ 3.151645] registered taskstats version 1 [ 3.152954] Loading compiled-in X.509 certificates [ 3.154393] zswap: loaded using pool lzo/zbud [ 3.171951] Key type big_key registered [ 3.182772] Key type encrypted registered [ 3.184544] ima: No TPM chip found, activating TPM-bypass! [ 3.186783] ima: Allocated hash algorithm: sha1 [ 3.188846] ima: No architecture policies found [ 3.190819] evm: Initialising EVM extended attributes: [ 3.192927] evm: security.selinux [ 3.194360] evm: security.ima [ 3.195418] evm: security.capability [ 3.196844] evm: HMAC attrs: 0x1 [ 3.199330] rtc_cmos 00:05: setting system clock to 2026-09-03 13:21:42 UTC (1788441702) [ 3.204947] debug: unmapping init [mem 0xffffffffada03000-0xffffffffadbfffff] [ 3.207671] debug: unmapping init [mem 0xffffffffac782000-0xffffffffaca58fff] [ 3.216233] Write protecting the kernel read-only data: 28672k [ 3.219952] debug: unmapping init [mem 0xffffffffaae03000-0xffffffffaaffffff] [ 3.223213] debug: unmapping init [mem 0xffffffffab714000-0xffffffffab7fffff] [ 3.257134] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.264397] systemd[1]: Detected virtualization kvm. [ 3.265875] systemd[1]: Detected architecture x86-64. [ 3.267465] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.295504] systemd[1]: No hostname configured. [ 3.297384] systemd[1]: Set hostname to . [ 3.299177] random: systemd: uninitialized urandom read (16 bytes read) [ 3.301270] systemd[1]: Initializing machine ID from random generator. [ 3.341600] random: ln: uninitialized urandom read (6 bytes read) [ 3.417991] random: systemd: uninitialized urandom read (16 bytes read) [ 3.420749] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.428576] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.432813] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.927646] device-mapper: uevent: version 1.0.3 [ 3.929704] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.548301] virtio_net virtio0 ens2: renamed from eth0 [ 4.558295] random: fast init done [ 4.684226] scsi host0: ata_piix [ 4.720267] scsi host1: ata_piix [ 4.721756] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.724224] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.854076] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.701178] random: crng init done [ 9.702496] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.942601] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.040528] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.305507] SELinux: Disabled at runtime. [ 11.365138] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.371492] systemd[1]: Detected virtualization kvm. [ 11.373500] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.817722] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.821330] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.825930] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.830207] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.834306] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.841864] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.849779] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ 11.906425] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Mounted /mnt. [ 12.190533] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ OK ] Started udev Coldplug all Devices. [ 12.434927] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.454917] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.570376] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.582205] EDAC sbridge: Ver: 1.1.2 [ 14.140272] Key type dns_resolver registered [ 14.430890] NFS: Registering the id_resolver key type [ 14.432859] Key type id_resolver registered [ 14.434414] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg155-server login: [ 40.552120] libcfs: loading out-of-tree module taints kernel. [ 40.573954] Key type ._llcrypt registered [ 40.575546] Key type .llcrypt registered [ 40.625285] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_hostid [ 59.832901] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 62.021262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.037574] alg: No test for adler32 (adler32-zlib) [ 63.746215] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 65.051811] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 66.961250] Key type lgssc registered [ 69.743669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 86.056350] hrtimer: interrupt took 5342072 ns [ 92.752394] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 140.715683] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 154.529115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 154.575658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 155.934228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 155.972757] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 156.105388] Lustre: lustre-MDT0000: new disk, initializing [ 156.271187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 156.307736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 161.901736] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.096251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.230610] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 177.277865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 177.286165] Lustre: Skipped 1 previous similar message [ 177.410930] Lustre: lustre-MDT0001: new disk, initializing [ 177.493492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 177.553384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 177.569567] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.332812] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.688635] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 199.369380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.687978] Lustre: lustre-OST0000: new disk, initializing [ 199.691427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.696645] Lustre: 8457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.762679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.354714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 205.369813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 205.524106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 207.281882] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.278298] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.576103] Lustre: lustre-OST0001: new disk, initializing [ 223.586088] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 223.597717] Lustre: 9530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 223.705988] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 231.705095] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 232.000503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 232.013627] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 232.104583] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 245.844230] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 253.781673] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 261.872279] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing check_logdir /tmp/testlogs/ [ 268.348471] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing yml_node [ 273.485314] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 276.394221] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 279.530417] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 281.407525] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:26:19 EDT 2026 [ 300.561419] Lustre: DEBUG MARKER: excepting tests: 21e 32b 32c 32newtarball [ 302.480546] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 319.464842] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.467915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 319.489538] Lustre: Skipped 1 previous similar message [ 319.506362] Lustre: Skipped 2 previous similar messages [ 324.579558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 324.585634] Lustre: Skipped 3 previous similar messages [ 329.711668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 329.719646] Lustre: Skipped 3 previous similar messages [ 333.282253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 333.468280] Lustre: server umount lustre-MDT0000 complete [ 338.192802] LustreError: 8456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442037 with bad export cookie 5727380605653627680 [ 338.194950] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 338.210442] LustreError: 8456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 339.937484] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.939122] LustreError: 7800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.940465] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 339.965803] Lustre: Skipped 3 previous similar messages [ 340.026399] LustreError: 7800:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 343.968878] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.974124] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.985167] LustreError: 6520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 343.985693] Lustre: Skipped 2 previous similar messages [ 344.807683] Lustre: server umount lustre-MDT0001 complete [ 348.998370] Lustre: server umount lustre-OST0000 complete [ 352.660567] Lustre: server umount lustre-OST0001 complete [ 359.962874] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_hostid [ 371.130239] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 414.673070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.002871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 415.045812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 415.174497] Lustre: lustre-MDT0000: new disk, initializing [ 415.294491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 415.317431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 420.497978] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 431.932463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.074632] Lustre: 15085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 432.130695] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 432.136993] Lustre: Skipped 1 previous similar message [ 432.241229] Lustre: lustre-MDT0001: new disk, initializing [ 432.474247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 432.521638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 432.529830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 438.697818] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 451.471626] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 458.953141] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 467.338950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 467.677464] Lustre: lustre-OST0000: new disk, initializing [ 467.686435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 467.694555] Lustre: 17057:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 467.840560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 469.388798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 469.408405] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 469.502897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 475.147464] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 486.269491] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 491.988559] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 492.353566] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 498.282751] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 498.555331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 505.318925] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.321590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 505.336081] Lustre: Skipped 1 previous similar message [ 505.336278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 505.357593] Lustre: Skipped 1 previous similar message [ 506.994509] Lustre: server umount lustre-OST0000 complete [ 514.547068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 514.562988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 517.675603] Lustre: server umount lustre-MDT0000 complete [ 520.679841] LustreError: 16313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.691846] LustreError: 16313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 521.943440] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442221 with bad export cookie 5727380605653629213 [ 521.951846] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 521.972964] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 522.313309] Lustre: server umount lustre-MDT0001 complete [ 525.997939] Lustre: DEBUG MARKER: == conf-sanity test 0: single mount setup ================ 09:30:24 (1788442224) [ 533.949152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 534.416994] LustreError: 19322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.481200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 539.624422] LustreError: 19323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.730606] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 548.836288] LustreError: 19322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.862533] LustreError: 19322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 549.081876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.626644] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 564.262277] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 570.431582] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 577.768048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 584.516147] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 593.207174] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 605.153945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 605.164112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 605.168339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 605.178191] Lustre: Skipped 2 previous similar messages [ 605.182110] Lustre: Skipped 3 previous similar messages [ 606.580341] Lustre: server umount lustre-OST0000 complete [ 616.479496] Lustre: server umount lustre-MDT0000 complete [ 619.907661] LustreError: 19303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442319 with bad export cookie 5727380605653630095 [ 619.915330] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 619.922233] LustreError: 19303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 620.583763] Lustre: server umount lustre-MDT0001 complete [ 635.350679] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 638.973457] Key type lgssc unregistered [ 639.416615] LNet: 22857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.424181] LNetError: 22857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 640.492267] LNet: Removed LNI 192.168.201.155@tcp [ 642.303284] Key type .llcrypt unregistered [ 642.305573] Key type ._llcrypt unregistered [ 667.449420] Key type ._llcrypt registered [ 667.453373] Key type .llcrypt registered [ 667.549629] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 668.943249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 668.962845] alg: No test for adler32 (adler32-zlib) [ 670.197326] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 670.489470] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 672.208193] Key type lgssc registered [ 673.484791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 681.979889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 682.002372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 683.509956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 689.090280] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 692.719314] LustreError: 24270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 692.742262] LustreError: 24270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 697.842119] LustreError: 24271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 697.925631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 698.249408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 702.740887] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 710.541095] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 716.612507] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 724.125880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 724.496132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 728.639910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 732.040955] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 739.414150] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 758.241922] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 758.246363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 758.251389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 758.263101] Lustre: Skipped 1 previous similar message [ 758.278448] Lustre: Skipped 1 previous similar message [ 763.368758] LustreError: 26093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 763.380057] LustreError: 26093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 763.473281] Lustre: server umount lustre-OST0000 complete [ 770.019594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 770.030343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 770.053879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 773.611022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 773.611457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 773.642070] Lustre: Skipped 1 previous similar message [ 778.721184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 778.732078] Lustre: Skipped 1 previous similar message [ 783.328350] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 783.554904] Lustre: server umount lustre-MDT0000 complete [ 783.866856] LustreError: 24270:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 783.886321] LustreError: 24270:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 787.445029] LustreError: 24252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442486 with bad export cookie 10283574493799801781 [ 787.452445] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 787.453721] LustreError: 24252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 787.663933] Lustre: server umount lustre-MDT0001 complete [ 797.226920] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 799.706435] Key type lgssc unregistered [ 799.987415] LNet: 28252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.001613] LNetError: 28252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 800.015693] LNet: Removed LNI 192.168.201.155@tcp [ 800.855493] Key type .llcrypt unregistered [ 800.862691] Key type ._llcrypt unregistered [ 823.954744] Key type ._llcrypt registered [ 823.957575] Key type .llcrypt registered [ 824.067148] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 825.016655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 825.135893] alg: No test for adler32 (adler32-zlib) [ 826.151791] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 826.383392] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 828.016170] Key type lgssc registered [ 829.210870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 836.791691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 836.815202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.189252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 842.601341] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 848.358704] LustreError: 29667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 848.382470] LustreError: 29667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 851.395827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 851.723764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 856.392583] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 864.574515] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 870.552245] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 884.988952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 885.380105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 891.955481] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 899.845150] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 907.885806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 920.802651] Lustre: server umount lustre-OST0000 complete [ 928.741139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 928.762688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 928.761733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 933.860529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 933.862693] Lustre: Skipped 2 previous similar messages [ 935.133064] Lustre: server umount lustre-MDT0000 complete [ 937.889870] Lustre: 28976:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442621/real 1788442621] req@000000009c16a2dc x1875317909126784/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788442637 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 937.917311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 937.931633] Lustre: Skipped 1 previous similar message [ 937.940514] LustreError: 30394:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 937.955678] LustreError: 30394:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 938.816540] LustreError: 29648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442638 with bad export cookie 15223478774191375419 [ 938.837800] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 938.841924] LustreError: 29648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 939.080062] Lustre: server umount lustre-MDT0001 complete [ 951.403792] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 954.971791] Key type lgssc unregistered [ 955.233748] LNet: 33571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.239225] LNetError: 33571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.251456] LNet: Removed LNI 192.168.201.155@tcp [ 955.769848] Key type .llcrypt unregistered [ 955.772080] Key type ._llcrypt unregistered [ 985.986561] Key type ._llcrypt registered [ 985.988357] Key type .llcrypt registered [ 986.316410] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 988.309403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 988.543619] alg: No test for adler32 (adler32-zlib) [ 989.540633] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 989.772745] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 991.645542] Key type lgssc registered [ 993.797480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1004.015182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1004.029409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1005.440894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1009.050152] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 1014.764793] LustreError: 34985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1014.797490] LustreError: 34985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1018.449617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1018.957501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1024.533323] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 1032.722856] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1039.295470] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1046.932942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1047.369828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1050.502401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1053.644614] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 1061.625549] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1075.177525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1075.181098] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.194447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1079.272300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1079.273186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.273569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1079.331735] Lustre: Skipped 1 previous similar message [ 1081.320107] Lustre: server umount lustre-OST0000 complete [ 1089.508933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.514930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1089.517059] Lustre: Skipped 1 previous similar message [ 1089.527451] Lustre: Skipped 1 previous similar message [ 1094.640050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1094.643978] Lustre: Skipped 1 previous similar message [ 1098.728839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1098.749736] Lustre: Skipped 1 previous similar message [ 1101.280215] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1101.607155] Lustre: server umount lustre-MDT0000 complete [ 1103.841929] LustreError: 34985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1103.870821] LustreError: 34985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1107.884240] LustreError: 34966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442807 with bad export cookie 8126688853392775666 [ 1107.899152] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1107.904930] LustreError: 34966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1108.629194] Lustre: server umount lustre-MDT0001 complete [ 1119.763640] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 1123.227001] Key type lgssc unregistered [ 1123.607455] LNet: 38549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1123.617985] LNetError: 38549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1124.658237] LNet: Removed LNI 192.168.201.155@tcp [ 1125.804979] Key type .llcrypt unregistered [ 1125.809172] Key type ._llcrypt unregistered [ 1153.237037] Key type ._llcrypt registered [ 1153.238891] Key type .llcrypt registered [ 1153.475911] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 1154.961072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1154.979351] alg: No test for adler32 (adler32-zlib) [ 1156.094947] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1156.326919] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 1157.984993] Key type lgssc registered [ 1159.923297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1168.448799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1168.469529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1169.865355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1174.504574] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 1180.134206] LustreError: 39970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1180.165613] LustreError: 39970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1184.206803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1184.652927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1189.725578] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 1199.407479] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1206.397839] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1213.724221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1213.958246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1215.035199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1221.360670] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 1228.694972] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1235.426115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1235.434477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.447705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1239.525907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1239.548136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1239.555425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.572390] Lustre: Skipped 2 previous similar messages [ 1240.714720] Lustre: lustre-OST0000: Not available for connect from 192.168.201.55@tcp (stopping) [ 1240.977651] Lustre: server umount lustre-OST0000 complete [ 1261.540951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1261.556253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1261.585205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1264.098040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.146355] Lustre: server umount lustre-MDT0000 complete [ 1269.218548] LustreError: 40962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.229190] LustreError: 40962:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1272.408366] LustreError: 39949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442971 with bad export cookie 16196988733251303911 [ 1272.422483] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1272.432896] LustreError: 39949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1278.927795] Lustre: server umount lustre-MDT0001 complete [ 1294.041286] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 1298.288945] Key type lgssc unregistered [ 1298.639685] LNet: 43683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.651201] LNetError: 43683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.662398] LNet: Removed LNI 192.168.201.155@tcp [ 1299.641369] Key type .llcrypt unregistered [ 1299.645261] Key type ._llcrypt unregistered [ 1330.432457] Key type ._llcrypt registered [ 1330.434161] Key type .llcrypt registered [ 1330.570775] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 1332.460305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1332.496311] alg: No test for adler32 (adler32-zlib) [ 1333.786087] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1334.143697] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 1335.880170] Key type lgssc registered [ 1337.678200] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1346.244924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1346.260517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1347.912022] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.414550] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 1358.309730] LustreError: 45099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1358.324082] LustreError: 45099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1360.611429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.042080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1365.153285] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 1373.069306] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1380.339958] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1387.300666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1387.790677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1392.892819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1393.947270] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 1402.957996] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1414.838613] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.55@tcp (stopping) [ 1416.676126] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1416.683160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1416.701673] Lustre: Skipped 1 previous similar message [ 1418.587830] Lustre: server umount lustre-MDT0000 complete [ 1419.923562] LustreError: 47589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1421.802034] LustreError: 46139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1421.829968] LustreError: 46139:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1423.006221] LustreError: 45080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443122 with bad export cookie 438755368568942594 [ 1423.020719] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1425.046137] LustreError: 45095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1425.061934] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.55@tcp (stopping) [ 1425.090386] Lustre: Skipped 2 previous similar messages [ 1426.916603] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1426.931889] Lustre: Skipped 1 previous similar message [ 1428.971951] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1428.987870] Lustre: Skipped 1 previous similar message [ 1429.522644] Lustre: server umount lustre-MDT0001 complete [ 1445.877199] Lustre: server umount lustre-OST0000 complete [ 1466.092523] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 1470.390179] Key type lgssc unregistered [ 1470.758162] LNet: 48964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1470.767486] LNetError: 48964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1470.783961] LNet: Removed LNI 192.168.201.155@tcp [ 1471.822348] Key type .llcrypt unregistered [ 1471.824070] Key type ._llcrypt unregistered [ 1499.508474] Key type ._llcrypt registered [ 1499.510490] Key type .llcrypt registered [ 1499.626547] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 1500.667028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1500.700860] alg: No test for adler32 (adler32-zlib) [ 1501.799979] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1502.107656] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 1503.824162] Key type lgssc registered [ 1504.964507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1514.631890] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1514.686754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1530.528832] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1535.648783] LustreError: 50351:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.155@tcp: failed processing log, type 4: rc = -110 [ 1562.405818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1568.746837] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 1577.195929] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1744.059321] Lustre: server umount lustre-OST0000 complete [ 1764.935245] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 1768.334176] Key type lgssc unregistered [ 1768.718801] LNet: 52030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1768.724166] LNetError: 52030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1768.741717] LNet: Removed LNI 192.168.201.155@tcp [ 1769.514105] Key type .llcrypt unregistered [ 1769.517243] Key type ._llcrypt unregistered [ 1796.150471] Key type ._llcrypt registered [ 1796.153895] Key type .llcrypt registered [ 1796.304757] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 1797.769859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1797.824372] alg: No test for adler32 (adler32-zlib) [ 1799.017471] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1799.320521] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 1801.018217] Key type lgssc registered [ 1802.622366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1812.812670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1812.832072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.438798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1820.266288] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 1823.732548] LustreError: 53446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1823.751751] LustreError: 53446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1828.837265] LustreError: 53447:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1830.493090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.892736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1834.891685] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 1843.816875] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1851.393763] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1859.705740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1860.296971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1866.917964] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 1875.780275] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1882.081681] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1882.097590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.111671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1884.198444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1885.396743] Lustre: server umount lustre-OST0000 complete [ 1889.251514] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1889.253571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1889.257577] Lustre: Skipped 1 previous similar message [ 1889.264510] Lustre: Skipped 2 previous similar messages [ 1894.369979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1894.383419] Lustre: Skipped 1 previous similar message [ 1895.410665] Lustre: server umount lustre-MDT0000 complete [ 1899.493869] LustreError: 53440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1899.514723] LustreError: 53440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1900.806107] LustreError: 53427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443600 with bad export cookie 13387754466972677282 [ 1900.825322] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1900.836501] LustreError: 53427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1901.130991] Lustre: server umount lustre-MDT0001 complete [ 1916.609799] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 1920.531387] Key type lgssc unregistered [ 1920.884607] LNet: 56982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.893386] LNetError: 56982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1921.963080] LNet: Removed LNI 192.168.201.155@tcp [ 1923.386100] Key type .llcrypt unregistered [ 1923.390079] Key type ._llcrypt unregistered [ 1949.862358] Key type ._llcrypt registered [ 1949.863926] Key type .llcrypt registered [ 1950.009525] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 1951.914111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1951.928560] alg: No test for adler32 (adler32-zlib) [ 1953.058602] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1953.290976] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 1954.961268] Key type lgssc registered [ 1956.197785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1964.984739] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1965.006653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1980.576538] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1985.697183] LustreError: 58371:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.155@tcp: failed processing log, type 4: rc = -110 [ 2012.389927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2019.882269] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2028.222285] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2036.235737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.827960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2036.912630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 2041.718087] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2041.826994] LustreError: 58395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2041.850063] LustreError: 58395:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2046.959406] LustreError: 59407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2046.978738] LustreError: 59407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2050.638538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.050459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2056.647744] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2065.143589] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2072.396081] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2076.649321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2076.652516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2076.658720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2076.664743] Lustre: Skipped 1 previous similar message [ 2081.082238] Lustre: server umount lustre-OST0000 complete [ 2103.361726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.380120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2103.385854] Lustre: Skipped 2 previous similar messages [ 2108.386313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2108.393454] Lustre: Skipped 1 previous similar message [ 2109.789517] Lustre: server umount lustre-MDT0000 complete [ 2113.506584] LustreError: 59412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2113.525034] LustreError: 59412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2114.280188] LustreError: 58378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443813 with bad export cookie 15406983160997937876 [ 2114.297291] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2114.300793] LustreError: 58378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2120.618224] Lustre: server umount lustre-MDT0001 complete [ 2132.063364] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 2135.123909] Key type lgssc unregistered [ 2135.356964] LNet: 62118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.365589] LNetError: 62118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2135.379392] LNet: Removed LNI 192.168.201.155@tcp [ 2136.231456] Key type .llcrypt unregistered [ 2136.233450] Key type ._llcrypt unregistered [ 2161.649803] Key type ._llcrypt registered [ 2161.655884] Key type .llcrypt registered [ 2161.812843] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 2162.787198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2162.823112] alg: No test for adler32 (adler32-zlib) [ 2164.107476] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2164.323094] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 2166.024259] Key type lgssc registered [ 2167.314583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2176.040301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2176.057195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.656641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2181.504702] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2187.247914] LustreError: 63531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2187.268475] LustreError: 63531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2190.612208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2190.954869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2195.771595] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2202.888532] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2209.819141] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2216.439213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2216.806928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2221.952915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 2223.075308] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2230.982358] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2251.768473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.773300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2251.802485] Lustre: Skipped 1 previous similar message [ 2251.803121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2254.750265] Lustre: server umount lustre-OST0000 complete [ 2261.998358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2262.011678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2262.015647] Lustre: Skipped 1 previous similar message [ 2262.033189] Lustre: Skipped 2 previous similar messages [ 2267.116065] LustreError: 63531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2267.137775] LustreError: 63531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2267.293171] Lustre: server umount lustre-MDT0000 complete [ 2271.803770] LustreError: 63513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443971 with bad export cookie 16921733826835697187 [ 2271.834149] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2272.497506] Lustre: server umount lustre-MDT0001 complete [ 2284.607649] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 2287.807769] Key type lgssc unregistered [ 2288.013982] LNet: 67091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2288.026157] LNetError: 67091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2288.065812] LNet: Removed LNI 192.168.201.155@tcp [ 2288.818233] Key type .llcrypt unregistered [ 2288.820614] Key type ._llcrypt unregistered [ 2324.347108] Key type ._llcrypt registered [ 2324.350752] Key type .llcrypt registered [ 2324.436187] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 2325.824626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2325.832469] alg: No test for adler32 (adler32-zlib) [ 2327.069948] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2327.592462] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 2329.233741] Key type lgssc registered [ 2330.826775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2340.652063] Lustre: DEBUG MARKER: == conf-sanity test 5h: start mdt failure at mdt_fs_setup() ========================================================== 10:00:39 (1788444039) [ 2347.697548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2347.728993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2349.093554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2353.794126] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2359.269438] LustreError: 68797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.290699] LustreError: 68797:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2364.391115] LustreError: 68798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2366.303291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2366.793543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2372.970679] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2383.606354] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2391.474786] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2401.115354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2401.713938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2409.256708] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2419.073681] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2419.213354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 2425.821282] Lustre: Failing over lustre-MDT0000 [ 2426.018440] LustreError: 70055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.475331] Lustre: server umount lustre-MDT0000 complete [ 2428.387197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2428.402343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.413578] LustreError: 68797:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.418908] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.548128] LustreError: 69538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2434.583176] LustreError: 69538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2437.862343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.156092] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2438.414340] Lustre: *** cfs_fail_loc=135, val=0*** [ 2438.533141] LustreError: 71902:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 2438.542308] LustreError: 71902:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.155@tcp: cfg command failed: rc = -2 [ 2438.555758] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2438.555758] [ 2438.566756] LustreError: MGC192.168.201.155@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2438.575283] LustreError: 71890:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 2438.580050] LustreError: 71890:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 2438.592878] LustreError: 71890:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2438.604616] LustreError: 71890:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2438.643413] Lustre: server umount lustre-MDT0000 complete [ 2438.650233] LustreError: 71890:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 2444.780494] LustreError: 68794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.809131] LustreError: 68794:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2445.608152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2445.874472] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2446.416775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2446.527778] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2451.432356] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2451.437419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2451.662944] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2451.754907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:321) [ 2451.927102] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2461.669382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2461.673030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2461.683548] Lustre: Skipped 2 previous similar messages [ 2466.312874] Lustre: server umount lustre-OST0000 complete [ 2471.911360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.939256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2471.953273] Lustre: Skipped 2 previous similar messages [ 2477.028341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2477.038569] Lustre: Skipped 1 previous similar message [ 2482.149205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2482.155499] Lustre: Skipped 1 previous similar message [ 2484.192379] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2484.476259] Lustre: server umount lustre-MDT0000 complete [ 2487.270260] LustreError: 70055:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2487.285846] LustreError: 70055:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2489.376542] LustreError: 69546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444188 with bad export cookie 13909588568328499095 [ 2489.386872] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2489.389624] LustreError: 69546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2489.927156] Lustre: server umount lustre-MDT0001 complete [ 2503.837739] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 2506.724836] Key type lgssc unregistered [ 2506.964500] LNet: 73685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2506.976993] LNetError: 73685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2507.020817] LNet: Removed LNI 192.168.201.155@tcp [ 2507.950984] Key type .llcrypt unregistered [ 2507.953133] Key type ._llcrypt unregistered [ 2533.965607] Key type ._llcrypt registered [ 2533.972355] Key type .llcrypt registered [ 2534.096615] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 2535.854314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2536.026551] alg: No test for adler32 (adler32-zlib) [ 2537.135221] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2537.475630] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 2539.280630] Key type lgssc registered [ 2541.310458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2551.536872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2551.553574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2553.253340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2558.553564] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2563.556544] LustreError: 75099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.572836] LustreError: 75099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2568.198589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2568.470148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2573.154538] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2580.429722] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2586.972982] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2596.611191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2596.987925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2598.030222] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:353) [ 2605.556696] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2617.803956] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2625.875463] Lustre: Failing over lustre-MDT0000 [ 2626.257987] LustreError: 77593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.183237] Lustre: server umount lustre-MDT0000 complete [ 2629.094134] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.106651] LustreError: 76605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2629.111777] Lustre: Skipped 2 previous similar messages [ 2629.139091] LustreError: 76605:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2631.319619] LustreError: 75095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2636.430079] LustreError: 75094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2636.466231] LustreError: 75094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2638.999858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2639.106274] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2639.182559] Lustre: *** cfs_fail_loc=a05, val=0*** [ 2639.262511] LustreError: 78190:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) [ 2639.268471] LustreError: 78190:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.155@tcp: cfg command failed: rc = -9 [ 2639.276567] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 2639.276567] [ 2639.289188] LustreError: MGC192.168.201.155@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 2639.295669] LustreError: 78178:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -9 [ 2639.301167] LustreError: 78178:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -9 [ 2639.308132] LustreError: 78178:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 2639.314883] LustreError: 78178:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 2639.335477] Lustre: server umount lustre-MDT0000 complete [ 2639.342323] LustreError: 78178:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -9 [ 2644.450934] LustreError: 75099:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2644.468591] LustreError: 75099:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2646.137045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2646.251131] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2646.500405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2646.545835] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2646.601829] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2651.634821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2651.658443] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2651.682621] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:385) [ 2651.736177] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2661.868930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.878475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2661.887775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2661.918622] Lustre: Skipped 1 previous similar message [ 2666.979704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2666.992099] Lustre: Skipped 1 previous similar message [ 2667.694661] Lustre: server umount lustre-OST0000 complete [ 2672.098247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.101311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2672.131810] Lustre: Skipped 2 previous similar messages [ 2672.153709] Lustre: Skipped 1 previous similar message [ 2677.223872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2677.229726] Lustre: Skipped 1 previous similar message [ 2678.471162] Lustre: server umount lustre-MDT0000 complete [ 2682.361896] LustreError: 76605:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.388476] LustreError: 76605:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2682.899316] LustreError: 75079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444382 with bad export cookie 7823162828274265631 [ 2682.906173] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2682.917935] LustreError: 75079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2683.181026] Lustre: server umount lustre-MDT0001 complete [ 2695.982151] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 2698.121555] Key type lgssc unregistered [ 2698.330703] LNet: 79974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2698.333720] LNetError: 79974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2698.343851] LNet: Removed LNI 192.168.201.155@tcp [ 2699.122665] Key type .llcrypt unregistered [ 2699.124198] Key type ._llcrypt unregistered [ 2713.673172] Key type ._llcrypt registered [ 2713.675334] Key type .llcrypt registered [ 2713.770304] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 2713.994589] Key type .llcrypt unregistered [ 2713.999092] Key type ._llcrypt unregistered [ 2741.259889] Key type ._llcrypt registered [ 2741.263442] Key type .llcrypt registered [ 2741.374896] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 2742.591851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2742.858588] alg: No test for adler32 (adler32-zlib) [ 2743.995496] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2744.392769] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 2746.152284] Key type lgssc registered [ 2748.083103] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2755.859313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2755.905815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2757.589657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2763.252969] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2766.818763] LustreError: 81881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.835541] LustreError: 81881:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2771.954036] LustreError: 81882:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2772.889121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2773.398477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2778.139658] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2787.082849] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2794.544159] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2801.944278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2802.149944] LustreError: 83703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2802.454355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2807.813076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:417) [ 2809.472540] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2818.235427] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2828.261550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2828.267377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2828.275446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2829.288208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2829.298029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2829.311995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2829.332964] Lustre: Skipped 1 previous similar message [ 2834.211288] Lustre: server umount lustre-OST0000 complete [ 2839.523246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2839.539745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2839.549315] Lustre: Skipped 1 previous similar message [ 2844.642525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2844.659894] Lustre: Skipped 1 previous similar message [ 2848.744894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2848.750657] Lustre: Skipped 1 previous similar message [ 2853.344134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2853.666769] Lustre: server umount lustre-MDT0000 complete [ 2853.858824] LustreError: 81881:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.882688] LustreError: 81881:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2858.248767] LustreError: 81863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444557 with bad export cookie 2776515360222992235 [ 2858.254092] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2858.263963] LustreError: 81863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2858.673822] Lustre: server umount lustre-MDT0001 complete [ 2870.733391] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 2873.140400] Key type lgssc unregistered [ 2873.611911] LNet: 85420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2873.627192] LNetError: 85420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2873.652911] LNet: Removed LNI 192.168.201.155@tcp [ 2874.655442] Key type .llcrypt unregistered [ 2874.656771] Key type ._llcrypt unregistered [ 2900.489488] Key type ._llcrypt registered [ 2900.492339] Key type .llcrypt registered [ 2900.626296] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 2901.990230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2902.021940] alg: No test for adler32 (adler32-zlib) [ 2903.083366] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2903.286292] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 2904.992179] Key type lgssc registered [ 2905.945848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2915.764795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2915.797329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2917.611328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.808531] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2927.082711] LustreError: 86835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2927.121805] LustreError: 86835:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2932.197427] LustreError: 86836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2933.896977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2934.229797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2939.818241] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2948.550992] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2955.177665] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2963.180586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2963.769424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2971.956905] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 2981.604803] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2987.389148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:449) [ 2992.809278] Lustre: server umount lustre-OST0000 complete [ 3000.810242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3000.815430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.822802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3002.851899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3002.859742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3007.974557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3007.980514] Lustre: Skipped 2 previous similar messages [ 3009.088886] Lustre: server umount lustre-MDT0000 complete [ 3010.021491] Lustre: 86146:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444693/real 1788444693] req@000000004962ae85 x1875320087016320/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788444709 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3010.066696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.105565] LustreError: 86835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3010.129146] LustreError: 86835:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3014.177209] Lustre: 86146:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444697/real 1788444697] req@000000007ec85329 x1875320087017344/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788444713 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3014.351927] LustreError: 86817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444713 with bad export cookie 2522048973746590271 [ 3014.355813] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3014.376796] LustreError: 86817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3014.844903] Lustre: server umount lustre-MDT0001 complete [ 3030.804708] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 3035.401553] Key type lgssc unregistered [ 3035.857662] LNet: 90375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3035.864354] LNetError: 90375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3035.883646] LNet: Removed LNI 192.168.201.155@tcp [ 3037.138519] Key type .llcrypt unregistered [ 3037.139895] Key type ._llcrypt unregistered [ 3067.155081] Key type ._llcrypt registered [ 3067.157143] Key type .llcrypt registered [ 3067.259893] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 3068.568646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3068.757109] alg: No test for adler32 (adler32-zlib) [ 3069.892515] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3070.209038] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 3071.944287] Key type lgssc registered [ 3073.149569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3082.421921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3082.443587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3083.842303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3089.785819] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 3093.986573] LustreError: 91795:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3094.016254] LustreError: 91795:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3099.118052] LustreError: 91796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.043968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3100.634548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3104.367268] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 3112.394343] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3119.853650] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3128.648706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3129.055681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3130.120883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:419 to 0x280000401:481) [ 3136.340448] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 3144.481601] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3155.938939] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3155.947304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.966818] LustreError: 93618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3156.246250] Lustre: server umount lustre-OST0000 complete [ 3162.099694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3162.110132] LustreError: Skipped 1 previous similar message [ 3162.112152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3162.133799] Lustre: Skipped 1 previous similar message [ 3162.137843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3166.178363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3166.182644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3167.753364] Lustre: server umount lustre-MDT0000 complete [ 3171.300397] LustreError: 91796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3171.318901] LustreError: 91796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3172.594258] LustreError: 92531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444871 with bad export cookie 11897350479781483092 [ 3172.612250] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3172.617914] LustreError: 92531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3173.148670] Lustre: server umount lustre-MDT0001 complete [ 3189.289800] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 3192.103372] Key type lgssc unregistered [ 3192.389691] LNet: 95344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3192.403981] LNetError: 95344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3192.426595] LNet: Removed LNI 192.168.201.155@tcp [ 3193.511569] Key type .llcrypt unregistered [ 3193.517585] Key type ._llcrypt unregistered [ 3223.204380] Key type ._llcrypt registered [ 3223.215390] Key type .llcrypt registered [ 3223.374406] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 3224.921891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3224.948043] alg: No test for adler32 (adler32-zlib) [ 3226.251890] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3226.720930] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 3228.570942] Key type lgssc registered [ 3230.306592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3239.292290] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3239.310213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3254.944466] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3260.064328] LustreError: 96738:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.155@tcp: failed processing log, type 4: rc = -110 [ 3286.752219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3293.901469] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 3304.241100] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3310.693783] Lustre: server umount lustre-OST0000 complete [ 3321.303166] Lustre: DEBUG MARKER: == conf-sanity test 10a: find lctl param broken symlinks ========================================================== 10:16:59 (1788445019) [ 3329.470417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3329.844817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3333.479437] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 3335.138397] LustreError: 98378:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3335.157365] LustreError: 98378:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3340.270623] LustreError: 98379:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3343.398078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3343.636201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3349.464817] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 3360.264735] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3366.889888] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3374.365518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3374.809413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3375.884671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:513) [ 3381.287305] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 3390.653758] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3406.817256] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3406.832464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.844976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3408.873487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3408.877658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3411.378670] Lustre: server umount lustre-OST0000 complete [ 3419.108822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.127467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3419.142188] Lustre: Skipped 2 previous similar messages [ 3422.533982] Lustre: server umount lustre-MDT0000 complete [ 3424.233556] LustreError: 99939:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3424.273932] LustreError: 99939:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3427.475782] LustreError: 98360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445126 with bad export cookie 12752969259629670577 [ 3427.489170] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3427.497751] LustreError: 98360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3428.039568] Lustre: server umount lustre-MDT0001 complete [ 3440.283733] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 3443.692553] Key type lgssc unregistered [ 3443.983727] LNet: 102085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3443.989785] LNetError: 102085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3444.017819] LNet: Removed LNI 192.168.201.155@tcp [ 3445.101146] Key type .llcrypt unregistered [ 3445.104657] Key type ._llcrypt unregistered [ 3493.910800] Key type ._llcrypt registered [ 3493.919045] Key type .llcrypt registered [ 3494.050390] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 3495.134719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3495.328654] alg: No test for adler32 (adler32-zlib) [ 3496.424433] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3496.653771] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 3498.336179] Key type lgssc registered [ 3499.786897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3509.520686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3509.535379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3511.179287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3516.759239] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 3521.506658] LustreError: 104271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3521.527928] LustreError: 104271:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3525.968537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.455632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3530.889114] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 3540.678685] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3548.334966] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3556.081147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3556.324319] LustreError: 106096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3556.569331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3561.969183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:484 to 0x280000401:545) [ 3564.583712] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 3573.634294] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3586.531321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3586.555296] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3586.566852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3586.596315] Lustre: Skipped 1 previous similar message [ 3589.552437] Lustre: server umount lustre-OST0000 complete [ 3596.773312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3596.789615] Lustre: Skipped 2 previous similar messages [ 3596.790708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3596.821354] Lustre: Skipped 1 previous similar message [ 3601.894413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3601.897039] Lustre: Skipped 1 previous similar message [ 3602.484969] Lustre: server umount lustre-MDT0000 complete [ 3605.987036] LustreError: 104271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3605.995721] LustreError: 104271:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3607.709537] LustreError: 104251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445307 with bad export cookie 14738887473499177536 [ 3607.737074] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3608.267988] Lustre: server umount lustre-MDT0001 complete [ 3620.583311] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 3623.964690] Key type lgssc unregistered [ 3624.364492] LNet: 107813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3624.374331] LNetError: 107813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3624.391393] LNet: Removed LNI 192.168.201.155@tcp [ 3625.314077] Key type .llcrypt unregistered [ 3625.319118] Key type ._llcrypt unregistered [ 3646.506494] Key type ._llcrypt registered [ 3646.508745] Key type .llcrypt registered [ 3646.661607] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 3648.073283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3648.137378] alg: No test for adler32 (adler32-zlib) [ 3649.180811] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3649.406165] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 3651.064219] Key type lgssc registered [ 3652.044646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3660.639395] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3660.652669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3676.256628] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3681.376365] LustreError: 109056:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.155@tcp: failed processing log, type 4: rc = -110 [ 3708.064968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3715.704455] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 3724.764191] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3732.006309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.280359] LustreError: MGC192.168.201.155@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 3732.289996] LustreError: 110080:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 3732.297866] LustreError: 110080:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 3732.302901] LustreError: 110080:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 3732.399296] Lustre: server umount lustre-MDT0000 complete [ 3732.402287] LustreError: 110080:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 3750.058544] Lustre: server umount lustre-OST0000 complete [ 3761.829195] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_hostid [ 3773.781267] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 3808.755069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.029594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3809.048515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3809.107387] Lustre: lustre-MDT0000: new disk, initializing [ 3809.213697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3809.237154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3813.863468] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 3824.279633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3824.376885] Lustre: 112875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3824.413328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3824.416468] Lustre: Skipped 1 previous similar message [ 3824.468125] Lustre: lustre-MDT0001: new disk, initializing [ 3824.524462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3824.551128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3824.559127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3829.452456] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 3839.471731] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3845.332379] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3852.554834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3852.856971] Lustre: lustre-OST0000: new disk, initializing [ 3852.859989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3852.866798] Lustre: 114846:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3852.958159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3858.781888] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 3860.590254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3860.613324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3860.760860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3866.572696] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3874.315911] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3874.793976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3881.107318] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3881.282770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3886.567450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3886.572928] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3886.644551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3886.684063] Lustre: Skipped 1 previous similar message [ 3891.706121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3891.716557] Lustre: Skipped 1 previous similar message [ 3896.805358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3896.810525] Lustre: Skipped 1 previous similar message [ 3898.337802] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3898.806793] Lustre: server umount lustre-OST0000 complete [ 3906.532647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3906.535788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3906.571688] Lustre: Skipped 1 previous similar message [ 3906.586267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3912.162072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3912.173512] Lustre: Skipped 3 previous similar messages [ 3919.330591] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3919.484479] Lustre: server umount lustre-MDT0000 complete [ 3923.701020] LustreError: 112866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445623 with bad export cookie 4710250081411336841 [ 3923.703770] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3923.716364] LustreError: 112866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3924.148209] Lustre: server umount lustre-MDT0001 complete [ 3932.541771] Lustre: DEBUG MARKER: == conf-sanity test 18: check mkfs creates large journals ========================================================== 10:27:11 (1788445631) [ 3936.141615] Lustre: DEBUG MARKER: use device /dev/mapper/mds1_flakey with MIN=2000000 [ 3952.691318] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_hostid [ 3962.563982] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 3999.758846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4000.051563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4000.086208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4000.145945] Lustre: lustre-MDT0000: new disk, initializing [ 4000.209545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4000.229211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4006.016840] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4018.409713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4018.510677] Lustre: 119748:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4018.577692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4018.586562] Lustre: Skipped 1 previous similar message [ 4018.685171] Lustre: lustre-MDT0001: new disk, initializing [ 4018.808731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4018.875065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4018.887034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4024.203619] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4034.664856] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4042.040226] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4050.115875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4050.466387] Lustre: lustre-OST0000: new disk, initializing [ 4050.478446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4050.490136] Lustre: 121718:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4050.628376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4057.547229] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4059.198791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4059.214372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4059.404576] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4068.674055] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4073.986900] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4074.326595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4079.896202] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4080.145439] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4085.237270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4085.257226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4085.261048] Lustre: Skipped 1 previous similar message [ 4085.274415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4085.276014] LustreError: Skipped 1 previous similar message [ 4085.288137] Lustre: Skipped 3 previous similar messages [ 4096.992134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4097.193414] Lustre: server umount lustre-OST0000 complete [ 4105.698674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4105.700878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4105.706513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4105.706523] Lustre: Skipped 4 previous similar messages [ 4105.720607] Lustre: Skipped 2 previous similar messages [ 4107.889485] Lustre: server umount lustre-MDT0000 complete [ 4110.822718] LustreError: 119760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4110.844118] LustreError: 119760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4112.218961] LustreError: 121716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445811 with bad export cookie 4710250081411337723 [ 4112.221151] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4112.244627] LustreError: 121716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4112.864048] Lustre: server umount lustre-MDT0001 complete [ 4122.866539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4123.603886] LustreError: 123936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4123.738310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4128.702878] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4128.746105] LustreError: 123937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4132.835241] LustreError: 123936:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4137.879901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4137.953541] LustreError: 123937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4143.601588] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4153.645933] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4160.130931] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4168.458102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4176.383979] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4184.484196] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4191.149903] Lustre: DEBUG MARKER: Success: mkfs creates large journals. Size: 78M [ 4195.809859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4195.812546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4195.818551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4195.823854] Lustre: Skipped 1 previous similar message [ 4197.173434] Lustre: server umount lustre-OST0000 complete [ 4205.028766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4205.030847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.054718] Lustre: Skipped 1 previous similar message [ 4207.956310] Lustre: server umount lustre-MDT0000 complete [ 4210.156776] LustreError: 124664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4210.175787] LustreError: 124664:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4212.030031] LustreError: 123916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445911 with bad export cookie 4710250081411338605 [ 4212.030808] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4212.047473] LustreError: 123916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4222.600803] Lustre: server umount lustre-MDT0001 complete [ 4235.374458] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 4238.613377] Key type lgssc unregistered [ 4238.868398] LNet: 127573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4238.881498] LNetError: 127573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4239.915137] LNet: Removed LNI 192.168.201.155@tcp [ 4240.984973] Key type .llcrypt unregistered [ 4240.988862] Key type ._llcrypt unregistered [ 4260.984440] Key type ._llcrypt registered [ 4260.989321] Key type .llcrypt registered [ 4261.099884] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_hostid [ 4277.651486] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 4278.977735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4279.167628] alg: No test for adler32 (adler32-zlib) [ 4280.331933] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4280.598404] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 4282.306658] Key type lgssc registered [ 4283.489620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4328.607383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4328.637811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4330.054086] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4330.145905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4330.334136] Lustre: lustre-MDT0000: new disk, initializing [ 4330.526229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4330.567498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4336.050618] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4348.187948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4348.447739] Lustre: 130728:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4348.524416] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4348.528323] Lustre: Skipped 1 previous similar message [ 4348.632396] Lustre: lustre-MDT0001: new disk, initializing [ 4348.737494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4348.816757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4348.831746] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4355.125367] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4368.515343] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4376.535904] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4385.510282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4386.135953] Lustre: lustre-OST0000: new disk, initializing [ 4386.152049] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4386.178801] Lustre: 132698:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4386.352492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4392.550345] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4392.557965] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4392.789421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4393.932202] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4404.633800] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4409.932968] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4410.163970] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4414.900454] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4415.106303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4418.535601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4418.539967] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4418.543152] LustreError: Skipped 1 previous similar message [ 4418.546466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4418.559838] Lustre: Skipped 1 previous similar message [ 4418.582936] Lustre: Skipped 1 previous similar message [ 4423.657745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4428.786727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4428.793112] Lustre: Skipped 2 previous similar messages [ 4431.328422] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4431.672035] Lustre: server umount lustre-OST0000 complete [ 4435.937391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4435.941533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4435.951929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4439.010967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4444.133167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4444.143545] Lustre: Skipped 3 previous similar messages [ 4450.272574] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4450.498756] Lustre: server umount lustre-MDT0000 complete [ 4454.569982] LustreError: 130721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446153 with bad export cookie 11523715301787689099 [ 4454.573045] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4454.582793] LustreError: 130721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4454.949756] Lustre: server umount lustre-MDT0001 complete [ 4463.610498] Lustre: DEBUG MARKER: == conf-sanity test 19a: start/stop MDS without OSTs ===== 10:36:02 (1788446162) [ 4471.172421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4471.619341] LustreError: 135112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4471.626050] LustreError: 135112:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4471.682117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4476.171581] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4476.916202] LustreError: 135113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4481.008148] LustreError: 135112:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4485.838941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4486.116833] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4486.119751] Lustre: Skipped 2 previous similar messages [ 4486.421869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4491.085443] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4502.187415] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4509.806431] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4516.322027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4516.340307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4516.342087] Lustre: Skipped 1 previous similar message [ 4516.358933] Lustre: Skipped 1 previous similar message [ 4521.165856] Lustre: server umount lustre-MDT0000 complete [ 4521.441631] LustreError: 135113:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4521.460984] LustreError: 135113:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4525.419824] LustreError: 135092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446224 with bad export cookie 11523715301787689981 [ 4525.434745] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4525.865382] Lustre: server umount lustre-MDT0001 complete [ 4534.538654] Lustre: DEBUG MARKER: == conf-sanity test 19b: start/stop OSTs without MDS ===== 10:37:13 (1788446233) [ 4540.719928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4556.257613] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4561.376578] LustreError: 137531:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.155@tcp: failed processing log, type 4: rc = -110 [ 4587.104381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4594.049357] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4602.040598] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4604.459056] Lustre: server umount lustre-OST0000 complete [ 4613.268333] Lustre: DEBUG MARKER: == conf-sanity test 20: remount ro,rw mounts work and doesn't break /etc/mtab ========================================================== 10:38:31 (1788446311) [ 4620.124582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4620.467770] LustreError: 138979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4620.511553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4625.461153] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4629.989395] LustreError: 138980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4630.023325] LustreError: 138980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4634.519994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4640.364449] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4649.172505] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4656.857287] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4664.544531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4664.805722] LustreError: 140800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4665.087126] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4665.106706] Lustre: Skipped 1 previous similar message [ 4672.170526] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4680.160592] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4691.425630] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4691.435809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4691.447631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4694.962290] Lustre: server umount lustre-MDT0000 complete [ 4697.060515] LustreError: 138974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4697.069375] LustreError: 138974:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4698.714302] LustreError: 138961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446398 with bad export cookie 11523715301787690646 [ 4698.727053] LustreError: 138961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4698.727933] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4699.241307] Lustre: server umount lustre-MDT0001 complete [ 4713.489739] Lustre: server umount lustre-OST0000 complete [ 4723.455856] Lustre: DEBUG MARKER: == conf-sanity test 21a: start mds before ost, stop ost first ========================================================== 10:40:21 (1788446421) [ 4730.993085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4731.619477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4736.976939] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4745.475538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4750.200140] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4758.760677] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4764.275515] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4770.949844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4776.319637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:65) [ 4777.081788] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4785.705258] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4791.573295] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4791.849458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4797.369688] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4797.629961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4802.025661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4802.027464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4802.042356] Lustre: Skipped 2 previous similar messages [ 4802.049902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4802.063629] Lustre: Skipped 3 previous similar messages [ 4805.951634] Lustre: server umount lustre-OST0000 complete [ 4816.542725] Lustre: server umount lustre-MDT0000 complete [ 4817.387405] LustreError: 142633:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4817.397176] LustreError: 142633:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4820.358954] LustreError: 143367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446519 with bad export cookie 11523715301787691717 [ 4820.368361] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4820.377028] LustreError: 143367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4821.029332] Lustre: server umount lustre-MDT0001 complete [ 4829.741849] Lustre: DEBUG MARKER: == conf-sanity test 21b: start ost before mds, stop mds first ========================================================== 10:42:08 (1788446528) [ 4837.634683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4853.218129] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 4858.336641] LustreError: 146792:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.155@tcp: failed processing log, type 4: rc = -110 [ 4884.064753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4884.075055] Lustre: Skipped 2 previous similar messages [ 4892.201799] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4900.355535] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4907.407801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4907.953090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:97) [ 4913.120094] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4921.761481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4927.040163] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 4934.413875] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4940.942280] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4945.980196] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4946.157441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4950.867278] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4951.073606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4958.186358] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4958.192587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4958.207311] Lustre: Skipped 4 previous similar messages [ 4958.209403] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4958.215672] Lustre: Skipped 4 previous similar messages [ 4959.866663] Lustre: server umount lustre-MDT0000 complete [ 4963.310961] LustreError: 149643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4963.355625] LustreError: 149643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4964.947801] LustreError: 146798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446664 with bad export cookie 11523715301787692739 [ 4964.948905] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4964.970178] LustreError: 146798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4965.641843] Lustre: server umount lustre-MDT0001 complete [ 4980.959322] Lustre: server umount lustre-OST0000 complete [ 4990.884414] Lustre: DEBUG MARKER: == conf-sanity test 21c: start mds between two osts, stop mds last ========================================================== 10:44:48 (1788446688) [ 4999.485942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5015.136647] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5020.258084] LustreError: 150989:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.155@tcp: failed processing log, type 4: rc = -110 [ 5052.596991] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5061.037172] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5068.847054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5069.316821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:129) [ 5073.716232] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5081.976767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5087.066656] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5095.296937] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5102.117482] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5108.805088] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5108.950883] Lustre: lustre-OST0001: new disk, initializing [ 5108.955068] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5108.962067] Lustre: 153862:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5115.463979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5115.476252] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5115.648981] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5116.020377] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5126.509950] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5131.551526] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5131.822772] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5136.875947] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5137.173155] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5139.430352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5139.443171] Lustre: Skipped 2 previous similar messages [ 5145.529724] Lustre: server umount lustre-OST0000 complete [ 5155.700827] Lustre: server umount lustre-OST0001 complete [ 5164.514419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5165.985657] Lustre: server umount lustre-MDT0000 complete [ 5169.566817] LustreError: 150996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446868 with bad export cookie 11523715301787693691 [ 5169.571667] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5169.580097] LustreError: 150996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5169.821872] Lustre: server umount lustre-MDT0001 complete [ 5193.192867] Lustre: DEBUG MARKER: == conf-sanity test 21d: start mgs then ost and then mds ========================================================== 10:48:11 (1788446891) [ 5194.565573] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21d need separate mgs device [ 5195.834571] Lustre: DEBUG MARKER: SKIP: conf-sanity test_21e skipping excluded test 21e [ 5197.407543] Lustre: DEBUG MARKER: == conf-sanity test 22: start a client before osts (should return errs) ========================================================== 10:48:15 (1788446895) [ 5204.792179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5205.195608] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5205.224756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5205.495568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5205.504386] Lustre: Skipped 6 previous similar messages [ 5209.974533] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5218.442036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5218.494545] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5218.512842] Lustre: 157424:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5222.810331] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5230.653607] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5236.428911] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5243.645048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5243.797963] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5251.105135] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5251.214565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:36 to 0x280000401:161) [ 5258.997670] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5264.224706] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5264.439609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5268.618914] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5268.840557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5271.521321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5271.528301] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5271.537296] Lustre: Skipped 4 previous similar messages [ 5271.542572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5271.546651] Lustre: Skipped 12 previous similar messages [ 5276.843897] Lustre: server umount lustre-OST0000 complete [ 5298.318484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5305.148700] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5312.047681] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5312.078569] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5312.145800] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5318.274258] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5318.585619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5322.351270] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5322.370477] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5323.124216] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5323.395488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5328.548746] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8f2a0708f800.ost_server_uuid 50 [ 5330.194230] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f2a0708f800.ost_server_uuid in FULL state after 0 sec [ 5330.397717] LustreError: 158521:0:(ldlm_lockd.c:727:ldlm_handle_ast_error()) ### client (nid 192.168.201.55@tcp) returned error from blocking AST (req@0000000022798dfb x1875321531146752 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: 0000000041bd8454/0x9fec7602ffc9fbd5 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.55@tcp remote: 0xad2e28e43a15b728 expref: 5 pid: 158521 timeout: 5430 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5330.437039] LustreError: lustre-MDT0000: A client on nid 192.168.201.55@tcp was evicted due to a lock blocking callback time out: rc -107 [ 5330.451400] LustreError: 157419:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 192.168.201.55@tcp ns: mdt-lustre-MDT0000_UUID lock: 0000000041bd8454/0x9fec7602ffc9fbd5 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 192.168.201.55@tcp remote: 0xad2e28e43a15b728 expref: 6 pid: 158521 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 [ 5336.044051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5336.050531] LustreError: Skipped 1 previous similar message [ 5340.600524] Lustre: server umount lustre-OST0000 complete [ 5350.741450] Lustre: server umount lustre-MDT0000 complete [ 5353.443178] LustreError: 158691:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5353.454727] LustreError: 158691:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 5354.664378] LustreError: 157416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447053 with bad export cookie 11523715301787694874 [ 5354.669230] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5354.670791] LustreError: 157416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5361.272474] Lustre: server umount lustre-MDT0001 complete [ 5372.158667] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 5374.663157] Key type lgssc unregistered [ 5374.895783] LNet: 163538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5374.899755] LNetError: 163538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5374.913805] LNet: Removed LNI 192.168.201.155@tcp [ 5375.443049] Key type .llcrypt unregistered [ 5375.444136] Key type ._llcrypt unregistered [ 5402.475493] Key type ._llcrypt registered [ 5402.477956] Key type .llcrypt registered [ 5402.604743] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 5404.520476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5404.602111] alg: No test for adler32 (adler32-zlib) [ 5405.757341] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5406.236478] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 5407.976159] Key type lgssc registered [ 5408.805822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5417.742241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5417.769402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5419.486403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5425.004704] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5428.711110] LustreError: 164953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5428.725462] LustreError: 164953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5433.827816] LustreError: 164954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5433.971959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5434.431782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5438.687105] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5447.230940] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5455.222440] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5464.203976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5464.618246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5471.614799] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5480.348440] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5489.261047] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5495.555345] Lustre: Failing over lustre-MDT0000 [ 5495.752937] Lustre: server umount lustre-MDT0000 complete [ 5495.780242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5495.784045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5495.801267] LustreError: 164954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5499.873259] LustreError: 165960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5499.876981] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5499.908475] LustreError: 165960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5499.923112] Lustre: Skipped 1 previous similar message [ 5504.994474] LustreError: 164954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5505.011165] LustreError: 164954:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5507.999209] LustreError: 164935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.55@tcp arrived at 1788447207 with bad export cookie 14822309770180631165 [ 5508.017509] LustreError: 164935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5515.020087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5515.159075] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5515.238873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 5515.249776] Lustre: Skipped 2 previous similar messages [ 5515.410805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5515.448573] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5520.257945] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5520.876394] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5520.883352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5522.831420] Lustre: lustre-MDT0000: Denying connection for new client 7463dd59-617a-4972-92e1-2d9e5ec8c13d (at 192.168.201.55@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:07 [ 5531.119222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5531.132104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5536.227806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5536.236919] Lustre: Skipped 2 previous similar messages [ 5537.308658] Lustre: server umount lustre-OST0000 complete [ 5541.130363] LustreError: 168709:0:(ldlm_lib.c:3004:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 5541.140872] Lustre: 168058:0:(ldlm_lib.c:2404:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 5541.146723] Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] [ 5541.158911] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] [ 5541.165700] LustreError: 168058:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009c4db1a7 x1875322711618304/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5541.179744] LustreError: 168058:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 5541.187302] LustreError: 168058:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 [ 5541.193567] Lustre: lustre-MDT0000: Recovery over after 0:21, of 2 clients 0 recovered and 2 were evicted. [ 5541.350952] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5541.367031] Lustre: Skipped 1 previous similar message [ 5541.378559] LustreError: 164948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5541.389965] LustreError: 164948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5541.498857] Lustre: server umount lustre-MDT0000 complete [ 5544.969824] LustreError: 164933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447244 with bad export cookie 14822309770180631494 [ 5544.973292] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5544.978724] LustreError: 164933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5544.996785] LustreError: 164261:0:(import.c:707:ptlrpc_connect_import_locked()) can't connect to a closed import [ 5547.341479] Lustre: server umount lustre-MDT0001 complete [ 5559.663799] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 5563.126796] Key type lgssc unregistered [ 5563.539233] LNet: 169464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5563.546292] LNetError: 169464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5563.568782] LNet: Removed LNI 192.168.201.155@tcp [ 5564.688678] Key type .llcrypt unregistered [ 5564.691253] Key type ._llcrypt unregistered [ 5591.424885] Key type ._llcrypt registered [ 5591.428797] Key type .llcrypt registered [ 5591.536820] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 5592.519352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5592.537989] alg: No test for adler32 (adler32-zlib) [ 5593.536392] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5593.774981] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 5595.536309] Key type lgssc registered [ 5596.940903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5605.117332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5605.132444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5606.587843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5611.296210] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5616.102236] LustreError: 170877:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5616.117143] LustreError: 170877:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5619.399195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5619.898639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5625.047743] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5633.477880] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5640.086408] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5647.607372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5647.995969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5651.059583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5655.106924] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5663.642595] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5670.886647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5670.888114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5670.897195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5670.927365] Lustre: Skipped 1 previous similar message [ 5672.173588] Lustre: server umount lustre-OST0000 complete [ 5676.003984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5676.015470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5676.015479] Lustre: Skipped 1 previous similar message [ 5676.024593] Lustre: Skipped 1 previous similar message [ 5681.123576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5681.133588] Lustre: Skipped 2 previous similar messages [ 5686.242213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5686.247525] Lustre: Skipped 1 previous similar message [ 5690.336134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5690.522336] Lustre: server umount lustre-MDT0000 complete [ 5691.364748] LustreError: 170872:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5691.381600] LustreError: 170872:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5694.472818] LustreError: 170859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447393 with bad export cookie 15029934448508455306 [ 5694.488497] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5695.009533] Lustre: server umount lustre-MDT0001 complete [ 5706.653953] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 5709.285523] Key type lgssc unregistered [ 5709.645636] LNet: 174413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5709.663723] LNetError: 174413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5710.708080] LNet: Removed LNI 192.168.201.155@tcp [ 5711.988837] Key type .llcrypt unregistered [ 5711.991487] Key type ._llcrypt unregistered [ 5743.152221] Key type ._llcrypt registered [ 5743.154223] Key type .llcrypt registered [ 5743.235191] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 5744.308782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5744.361811] alg: No test for adler32 (adler32-zlib) [ 5745.534024] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5745.831357] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 5747.448208] Key type lgssc registered [ 5748.421668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5756.548784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5756.567673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5758.046137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5763.596605] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5768.164067] LustreError: 176109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5768.175437] LustreError: 176109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5773.284876] LustreError: 176108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5773.973646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5774.346275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5779.812365] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5788.177943] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5794.139528] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5801.325633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5801.744341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5802.805964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 5807.793746] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5814.598665] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5828.580838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5828.587845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5828.604960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5829.601931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5829.605673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5829.621242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5829.677458] Lustre: Skipped 1 previous similar message [ 5833.980428] Lustre: server umount lustre-OST0000 complete [ 5839.864926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5839.872148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5839.884948] Lustre: Skipped 1 previous similar message [ 5844.963160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5844.982652] Lustre: Skipped 2 previous similar messages [ 5850.089872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5850.099044] Lustre: Skipped 1 previous similar message [ 5852.128131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5852.291747] Lustre: server umount lustre-MDT0000 complete [ 5855.217078] LustreError: 176109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5855.236923] LustreError: 176109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5856.255842] LustreError: 176090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447555 with bad export cookie 12421763142778954974 [ 5856.261499] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5856.266947] LustreError: 176090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5856.790313] Lustre: server umount lustre-MDT0001 complete [ 5870.130782] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 5874.114643] Key type lgssc unregistered [ 5874.422149] LNet: 179671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5874.433537] LNetError: 179671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5874.459187] LNet: Removed LNI 192.168.201.155@tcp [ 5875.696159] Key type .llcrypt unregistered [ 5875.697916] Key type ._llcrypt unregistered [ 5902.235151] Key type ._llcrypt registered [ 5902.238748] Key type .llcrypt registered [ 5902.357180] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 5903.227693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5903.242463] alg: No test for adler32 (adler32-zlib) [ 5904.287776] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5904.597521] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 5906.312203] Key type lgssc registered [ 5907.297744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5918.656147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5918.679606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.094070] Lustre: *** cfs_fail_loc=135, val=0*** [ 5920.167533] LustreError: 181198:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) [ 5920.178472] LustreError: 181198:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.201.155@tcp: cfg command failed: rc = -2 [ 5920.200205] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 5920.200205] [ 5920.227141] LustreError: MGC192.168.201.155@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 5920.243195] LustreError: 181159:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-MDT0000: -2 [ 5920.257159] LustreError: 181159:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 5920.272932] LustreError: 181159:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-MDT0000 [ 5920.279701] LustreError: 181159:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 5920.342470] Lustre: server umount lustre-MDT0000 complete [ 5920.344225] LustreError: 181159:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5928.122343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5928.835505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5934.028730] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5939.169353] LustreError: 181568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5939.176367] LustreError: 181568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5943.083452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5943.666598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5948.460656] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 5955.539753] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5961.726660] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5964.261226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5964.266351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5964.272295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5968.418677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5968.424028] Lustre: Skipped 2 previous similar messages [ 5970.663669] Lustre: server umount lustre-MDT0000 complete [ 5973.473648] LustreError: 181568:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5973.490423] LustreError: 181568:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5974.049442] LustreError: 181550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447673 with bad export cookie 12752346070531061557 [ 5974.061129] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5974.064279] LustreError: 181550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5974.307263] Lustre: server umount lustre-MDT0001 complete [ 5985.600559] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 5988.992064] Key type lgssc unregistered [ 5989.223126] LNet: 183903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5989.226261] LNetError: 183903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5989.238739] LNet: Removed LNI 192.168.201.155@tcp [ 5990.015450] Key type .llcrypt unregistered [ 5990.020159] Key type ._llcrypt unregistered [ 6010.634038] Key type ._llcrypt registered [ 6010.636286] Key type .llcrypt registered [ 6010.736867] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 6010.988314] Key type .llcrypt unregistered [ 6010.991099] Key type ._llcrypt unregistered [ 6027.462604] Key type ._llcrypt registered [ 6027.464911] Key type .llcrypt registered [ 6027.566876] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 6028.689463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6028.713643] alg: No test for adler32 (adler32-zlib) [ 6029.912274] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6030.232606] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 6031.944714] Key type lgssc registered [ 6033.263689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6041.320614] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6041.341077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6056.928679] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6062.048335] LustreError: 186039:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.155@tcp: failed processing log, type 4: rc = -110 [ 6088.736213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6095.069292] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6103.441588] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6111.043744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6111.776596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6111.897682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 6116.838575] LustreError: 186064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6116.869018] LustreError: 186064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6117.168685] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6121.965318] LustreError: 186065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6122.985143] LustreError: 186064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6122.998818] LustreError: 186064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6127.158497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6127.557831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6131.996735] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6139.903400] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6147.127959] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6150.167411] Lustre: Setting parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6162.916719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6162.929704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.944281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6163.433719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6163.437662] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6163.464416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6163.467161] Lustre: Skipped 1 previous similar message [ 6168.504795] Lustre: server umount lustre-OST0000 complete [ 6173.668599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6173.674750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6173.688946] Lustre: Skipped 1 previous similar message [ 6173.699779] Lustre: Skipped 1 previous similar message [ 6178.785681] LustreError: 187080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6178.805179] LustreError: 187080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6178.889644] Lustre: server umount lustre-MDT0000 complete [ 6182.596752] LustreError: 186045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447881 with bad export cookie 11458374356437143105 [ 6182.602728] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6182.609346] LustreError: 186045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6182.859689] Lustre: server umount lustre-MDT0001 complete [ 6193.081330] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 6196.259973] Key type lgssc unregistered [ 6196.535657] LNet: 190056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6196.538519] LNetError: 190056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6196.573391] LNet: Removed LNI 192.168.201.155@tcp [ 6197.410647] Key type .llcrypt unregistered [ 6197.413294] Key type ._llcrypt unregistered [ 6221.813569] Key type ._llcrypt registered [ 6221.817137] Key type .llcrypt registered [ 6221.964298] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 6222.768531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6222.784155] alg: No test for adler32 (adler32-zlib) [ 6223.791239] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6224.044438] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 6225.744280] Key type lgssc registered [ 6227.053987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6235.304822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6235.319936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6236.939405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6244.298460] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6247.397482] LustreError: 191468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6247.420291] LustreError: 191468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6252.514138] LustreError: 191469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6254.399666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6255.272028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6261.348815] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6269.613432] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6275.501303] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6282.956145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6283.302405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6289.016524] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6295.294808] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6306.891213] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:321) [ 6312.265524] Lustre: Failing over lustre-MDT0000 [ 6312.616032] LustreError: 191464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6312.657760] Lustre: server umount lustre-MDT0000 complete [ 6316.517197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6316.523514] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6316.552837] LustreError: 192198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6317.034224] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6322.152792] LustreError: 193313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6322.177132] LustreError: 193313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6331.473567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6331.569201] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6331.856189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6331.908639] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6332.939219] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6335.472353] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6337.008892] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6337.019758] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 6337.053558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:353) [ 6339.715291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_acquire_expire=35 in log lustre-MDT0000 [ 6348.061342] Lustre: Setting parameter lustre-MDT0000-mdc.mdc.max_rpcs_in_flight=13 in log lustre-client [ 6362.082030] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6362.088281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6362.109795] Lustre: Skipped 1 previous similar message [ 6362.120154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6367.089404] Lustre: server umount lustre-OST0000 complete [ 6372.833449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6372.838829] LustreError: Skipped 1 previous similar message [ 6372.849932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6372.857408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6372.857418] Lustre: Skipped 1 previous similar message [ 6372.904460] Lustre: Skipped 2 previous similar messages [ 6377.955044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6377.959363] Lustre: Skipped 2 previous similar messages [ 6383.074447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6383.080501] Lustre: Skipped 1 previous similar message [ 6385.120332] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6385.362408] Lustre: server umount lustre-MDT0000 complete [ 6388.198566] LustreError: 192198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6388.213853] LustreError: 192198:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6390.046171] LustreError: 191450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448089 with bad export cookie 5323968943986163324 [ 6390.050964] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6390.055569] LustreError: 191450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6390.548244] Lustre: server umount lustre-MDT0001 complete [ 6402.164639] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 6405.201407] Key type lgssc unregistered [ 6405.480201] LNet: 196403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6405.493631] LNetError: 196403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6405.530992] LNet: Removed LNI 192.168.201.155@tcp [ 6406.527774] Key type .llcrypt unregistered [ 6406.529897] Key type ._llcrypt unregistered [ 6428.369089] Key type ._llcrypt registered [ 6428.372128] Key type .llcrypt registered [ 6428.475134] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 6429.324730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6429.387680] alg: No test for adler32 (adler32-zlib) [ 6430.447658] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6430.642054] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 6432.344227] Key type lgssc registered [ 6433.536096] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6440.762221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6440.791099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6442.144282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6446.225187] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6452.714077] LustreError: 197817:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6452.739873] LustreError: 197817:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6454.215818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6454.526073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6460.300326] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6468.191506] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6474.880809] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6481.832599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6482.063790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6486.965613] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6488.272434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:385) [ 6494.133530] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6496.750526] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=5 in log lustre-client [ 6504.691864] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=6 in log lustre-client [ 6518.536599] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 6525.417767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6525.423551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6525.429446] Lustre: Skipped 1 previous similar message [ 6525.441685] Lustre: Skipped 1 previous similar message [ 6530.532741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6530.537671] Lustre: Skipped 1 previous similar message [ 6530.912785] Lustre: server umount lustre-OST0000 complete [ 6535.651532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6535.668307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6535.674398] Lustre: Skipped 1 previous similar message [ 6540.622370] Lustre: server umount lustre-MDT0000 complete [ 6540.769491] LustreError: 198546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6540.804320] LustreError: 198546:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6544.827063] LustreError: 197797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448244 with bad export cookie 10528749125441688961 [ 6544.830728] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6544.834696] LustreError: 197797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6545.104979] Lustre: server umount lustre-MDT0001 complete [ 6555.291236] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 6558.700506] Key type lgssc unregistered [ 6559.140812] LNet: 201671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6559.150404] LNetError: 201671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6560.167329] LNet: Removed LNI 192.168.201.155@tcp [ 6561.272598] Key type .llcrypt unregistered [ 6561.278786] Key type ._llcrypt unregistered [ 6580.392694] Key type ._llcrypt registered [ 6580.395973] Key type .llcrypt registered [ 6580.521798] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 6581.283353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6581.336276] alg: No test for adler32 (adler32-zlib) [ 6582.577476] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6582.861358] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 6584.689210] Key type lgssc registered [ 6585.904130] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6595.584505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6595.606681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6597.101196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6601.673610] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6607.332771] LustreError: 202937:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6607.349344] LustreError: 202937:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6611.151470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6611.607234] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6616.924392] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6625.300672] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6631.183614] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6638.906887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6639.251705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6642.363898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:417) [ 6645.770776] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6653.190625] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6660.012725] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=230 in log lustre-OST0000 [ 6670.314607] Lustre: Modifying parameter lustre-OST0000.ost.client_cache_seconds=115 in log lustre-OST0000 [ 6678.950135] Lustre: Setting parameter lustre-OST0000.osd.auto_scrub=1 in log lustre-OST0000 [ 6685.912621] Lustre: Modifying parameter lustre-OST0000.osd.auto_scrub=2592000 in log lustre-OST0000 [ 6697.961340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6698.009356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6698.023261] Lustre: Skipped 1 previous similar message [ 6700.721771] Lustre: server umount lustre-OST0000 complete [ 6708.195107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6708.200954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6708.211660] Lustre: Skipped 2 previous similar messages [ 6712.410218] Lustre: server umount lustre-MDT0000 complete [ 6713.322944] LustreError: 204289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6713.345567] LustreError: 204289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6717.902756] LustreError: 202916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448417 with bad export cookie 8614158681508641699 [ 6717.902756] LustreError: 205616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448417 with bad export cookie 8614158681508641699 [ 6717.903812] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6717.917879] LustreError: 205616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6718.517784] Lustre: server umount lustre-MDT0001 complete [ 6729.892581] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 6732.985590] Key type lgssc unregistered [ 6733.277698] LNet: 207673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6733.296702] LNetError: 207673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6734.319605] LNet: Removed LNI 192.168.201.155@tcp [ 6735.147556] Key type .llcrypt unregistered [ 6735.152657] Key type ._llcrypt unregistered [ 6759.750098] Key type ._llcrypt registered [ 6759.755515] Key type .llcrypt registered [ 6759.896892] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 6761.229368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6761.271316] alg: No test for adler32 (adler32-zlib) [ 6762.325232] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6762.538480] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 6764.168234] Key type lgssc registered [ 6765.284515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6773.055891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6773.084145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6774.486590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6779.389119] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6783.983628] LustreError: 209090:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6784.009859] LustreError: 209090:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6788.434246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6788.695996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6793.430800] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6801.478313] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6807.715335] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6814.709594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6814.982357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6819.148270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:449) [ 6821.059567] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6828.508766] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6843.874530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6843.880184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6843.898579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6843.902198] Lustre: Skipped 1 previous similar message [ 6848.499715] Lustre: server umount lustre-OST0000 complete [ 6852.066185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6852.068688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6852.082333] Lustre: Skipped 2 previous similar messages [ 6852.097618] Lustre: Skipped 1 previous similar message [ 6854.115345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6854.125067] Lustre: Skipped 1 previous similar message [ 6859.234981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6859.240794] Lustre: Skipped 1 previous similar message [ 6864.353939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6864.368935] Lustre: Skipped 1 previous similar message [ 6866.400812] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6866.544326] Lustre: server umount lustre-MDT0000 complete [ 6869.474827] LustreError: 209085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6869.499861] LustreError: 209085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6870.211692] LustreError: 209070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448569 with bad export cookie 12031911628219412727 [ 6870.222208] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6870.230808] LustreError: 209070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6870.647812] Lustre: server umount lustre-MDT0001 complete [ 6881.416866] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 6884.006753] Key type lgssc unregistered [ 6884.269308] LNet: 212777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6884.282215] LNetError: 212777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6884.313052] LNet: Removed LNI 192.168.201.155@tcp [ 6885.276171] Key type .llcrypt unregistered [ 6885.279391] Key type ._llcrypt unregistered [ 6902.950592] Key type ._llcrypt registered [ 6902.951843] Key type .llcrypt registered [ 6903.076275] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 6904.103695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6904.143890] alg: No test for adler32 (adler32-zlib) [ 6905.199320] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6905.421787] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 6907.069200] Key type lgssc registered [ 6908.313039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6915.714194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6915.724100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6917.310175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6921.270424] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6926.840141] LustreError: 214044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6926.859318] LustreError: 214044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6930.102483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.477060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6935.173932] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6943.870206] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6950.956395] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6958.343475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6958.842416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6961.940297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:481) [ 6965.524408] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 6973.764415] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6987.745846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6987.758093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6987.766216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6990.822972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6990.826411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6990.827186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6990.833458] Lustre: Skipped 1 previous similar message [ 6993.333651] Lustre: server umount lustre-OST0000 complete [ 7001.057345] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7001.060374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7001.071643] Lustre: Skipped 1 previous similar message [ 7003.913934] Lustre: server umount lustre-MDT0000 complete [ 7006.184803] LustreError: 214038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7006.203179] LustreError: 214038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7007.535855] LustreError: 215889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448706 with bad export cookie 703760177684858813 [ 7007.545881] LustreError: 215889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7007.548831] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7007.843506] Lustre: server umount lustre-MDT0001 complete [ 7019.184164] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 7021.607019] Key type lgssc unregistered [ 7021.870774] LNet: 217731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7021.879878] LNetError: 217731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7021.901979] LNet: Removed LNI 192.168.201.155@tcp [ 7023.025068] Key type .llcrypt unregistered [ 7023.028061] Key type ._llcrypt unregistered [ 7042.903736] Key type ._llcrypt registered [ 7042.906070] Key type .llcrypt registered [ 7043.043756] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 7044.653882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7044.739473] alg: No test for adler32 (adler32-zlib) [ 7045.864373] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7046.179785] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 7047.872286] Key type lgssc registered [ 7049.173084] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7057.095638] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7057.113213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7058.635037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7063.012532] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 7068.137564] LustreError: 218998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7068.157432] LustreError: 218998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7072.004896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7072.422845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7077.265272] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 7084.926826] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7090.793732] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7098.417986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7098.844399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7102.957823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:513) [ 7105.337776] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 7113.520429] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7124.245638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7124.295700] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7124.458140] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7130.910290] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 7139.194172] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7140.801887] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7152.492912] Lustre: Permanently deactivating lustre-OST0001 [ 7152.531882] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7160.910637] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 7167.268400] Lustre: Permanently reactivating lustre-OST0001 [ 7167.286483] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7167.291758] Lustre: Skipped 2 previous similar messages [ 7170.052741] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7170.067359] Lustre: lustre-OST0001: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7170.077597] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7170.100210] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7170.158763] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7180.258590] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7180.268197] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7180.284294] Lustre: Skipped 1 previous similar message [ 7180.297655] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7180.769841] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7180.912474] Lustre: server umount lustre-OST0001 complete [ 7183.332945] LustreError: 220822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7183.344879] LustreError: 220822:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7188.459819] LustreError: 220822:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7188.464095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7188.472587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7188.472599] Lustre: Skipped 1 previous similar message [ 7188.486359] LustreError: 220822:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7188.523181] Lustre: Skipped 2 previous similar messages [ 7191.367656] Lustre: server umount lustre-OST0000 complete [ 7198.692276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7198.698078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7198.712507] Lustre: Skipped 1 previous similar message [ 7198.723871] Lustre: Skipped 1 previous similar message [ 7201.957797] Lustre: server umount lustre-MDT0000 complete [ 7203.811508] LustreError: 218998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7203.848778] LustreError: 218998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7206.365071] LustreError: 222621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448905 with bad export cookie 15494200631259119544 [ 7206.371803] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7206.383220] LustreError: 222621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7206.786607] Lustre: server umount lustre-MDT0001 complete [ 7217.847932] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 7221.039945] Key type lgssc unregistered [ 7221.291648] LNet: 224050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7221.300509] LNetError: 224050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7221.318757] LNet: Removed LNI 192.168.201.155@tcp [ 7222.420111] Key type .llcrypt unregistered [ 7222.422207] Key type ._llcrypt unregistered [ 7264.353965] Key type ._llcrypt registered [ 7264.356917] Key type .llcrypt registered [ 7264.530581] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 7265.764915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7265.911784] alg: No test for adler32 (adler32-zlib) [ 7266.944576] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7267.186446] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 7268.849901] Key type lgssc registered [ 7269.894101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7278.472241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7278.491224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7279.949214] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7279.968621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7280.104794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7284.929915] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 7294.071193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7294.159659] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7294.204311] Lustre: 226256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7294.494845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7299.129736] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 7306.847518] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7313.276944] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7320.805141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7321.102964] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7321.459063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7326.924725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:545) [ 7329.456734] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 7336.201494] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7339.372447] Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7346.839505] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=2 in log lustre-client [ 7355.967986] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=3 in log lustre-client [ 7364.776502] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7380.955365] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7380.968212] Lustre: Skipped 1 previous similar message [ 7404.096604] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=1 in log lustre-client [ 7404.105296] Lustre: Skipped 2 previous similar messages [ 7442.837520] Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb=4 in log lustre-client [ 7442.842346] Lustre: Skipped 4 previous similar messages [ 7508.080279] Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb= in log lustre-client [ 7508.086700] Lustre: Skipped 7 previous similar messages [ 7516.662641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7516.671652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7516.679509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7516.694745] Lustre: Skipped 1 previous similar message [ 7516.714611] Lustre: Skipped 1 previous similar message [ 7521.503974] Lustre: server umount lustre-OST0000 complete [ 7527.523117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7527.533244] Lustre: Skipped 1 previous similar message [ 7527.535677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7527.555208] Lustre: Skipped 1 previous similar message [ 7533.031399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7533.047383] Lustre: Skipped 1 previous similar message [ 7533.969385] Lustre: server umount lustre-MDT0000 complete [ 7539.155620] LustreError: 229047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449238 with bad export cookie 4786173622187187068 [ 7539.165674] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7539.178609] LustreError: 229047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7539.611659] Lustre: server umount lustre-MDT0001 complete [ 7552.962732] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 7557.523543] Key type lgssc unregistered [ 7558.199432] LNet: 231054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7558.215251] LNetError: 231054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7558.253271] LNet: Removed LNI 192.168.201.155@tcp [ 7559.391141] Key type .llcrypt unregistered [ 7559.394168] Key type ._llcrypt unregistered [ 7586.389586] Key type ._llcrypt registered [ 7586.395231] Key type .llcrypt registered [ 7586.575091] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 7588.186175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7588.416863] alg: No test for adler32 (adler32-zlib) [ 7589.584515] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7589.926688] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 7591.736215] Key type lgssc registered [ 7593.197982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7601.017744] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7601.032666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7602.539882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7607.194342] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 7611.887139] LustreError: 232472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7611.902089] LustreError: 232472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7615.479850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7615.942884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7620.345619] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 7627.253725] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7633.769486] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7640.429181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7640.751664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7646.908187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:577) [ 7648.716325] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_default_debug -1 all [ 7658.641508] Lustre: DEBUG MARKER: oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7677.900711] Lustre: server umount lustre-OST0000 complete [ 7682.532152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7682.540458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7682.566160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7683.041699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7688.172522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7688.180210] Lustre: Skipped 3 previous similar messages [ 7692.359185] Lustre: server umount lustre-MDT0000 complete [ 7693.282056] LustreError: 233488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7693.300631] LustreError: 233488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7693.859428] Lustre: 231783:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449377/real 1788449377] req@00000000c872a374 x1875325001698304/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788449393 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7693.888580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7696.849587] LustreError: 232454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449396 with bad export cookie 5799529477375373445 [ 7696.859912] LustreError: MGC192.168.201.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7696.880091] LustreError: 232454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7697.513668] Lustre: server umount lustre-MDT0001 complete [ 7710.802424] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 7713.869319] Key type lgssc unregistered [ 7714.207924] LNet: 236167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7714.213319] LNetError: 236167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7714.223784] LNet: Removed LNI 192.168.201.155@tcp [ 7715.148787] Key type .llcrypt unregistered [ 7715.150410] Key type ._llcrypt unregistered [ 7740.521395] Key type ._llcrypt registered [ 7740.523631] Key type .llcrypt registered [ 7740.640622] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 7742.459788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7742.569863] alg: No test for adler32 (adler32-zlib) [ 7743.706507] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7743.958527] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 7745.601467] Key type lgssc registered [ 7746.936962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8023.745854] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 8027.051040] Key type lgssc unregistered [ 8027.477735] LNet: 238196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8027.483388] LNetError: 238196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8027.528532] LNet: Removed LNI 192.168.201.155@tcp [ 8028.574319] Key type .llcrypt unregistered [ 8028.575920] Key type ._llcrypt unregistered [ 8060.636491] Key type ._llcrypt registered [ 8060.637588] Key type .llcrypt registered [ 8060.703683] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing set_hostid [ 8076.191558] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing load_modules_local [ 8076.973643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8077.047252] alg: No test for adler32 (adler32-zlib) [ 8078.062735] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8078.291906] LNet: Added LNI 192.168.201.155@tcp [8/256/0/180] [ 8079.976699] Key type lgssc registered [ 8081.313744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8118.854367] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7835 sec ========== 11:36:56 (1788449816) [ 8120.744303] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:36:58 (1788449818) === [ 8124.887687] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:37:02 (1788449822) === [ 8167.476979] Lustre: DEBUG MARKER: oleg155-server.virtnet: executing unload_modules_local [ 8170.494773] Key type lgssc unregistered [ 8170.792471] LNet: 243440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8170.805415] LNetError: 243440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8170.819669] LNet: Removed LNI 192.168.201.155@tcp [ 8171.601205] Key type .llcrypt unregistered [ 8171.604861] Key type ._llcrypt unregistered